Spark HBaseTest不工作

Question

尝试使用kerberos auth在集群模式下对yarn上的HBaseTest进行spark设置-

$SPARK_HOME/bin/spark-submit \
--master yarn-cluster \
--class org.apache.spark.examples.HBaseTest \
--jars /apache/hbase/lib/*.jars \
--driver-class-path /apache/hbase/conf \
$SPARK_HOME/lib/spark-examples.jar \
myhbasetable

我最终得到了-

User class threw exception: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
Tue Jan 19 01:00:22 GMT-07:00 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@6515a0f9, java.io.IOException: Failed to find location, tableName=hbase:meta, row=myhbasetable,,00000000000000, reload=false
Tue Jan 19 01:00:50 GMT-07:00 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@6515a0f9, java.io.IOException: Enable/Disable failed
Tue Jan 19 01:01:16 GMT-07:00 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@6515a0f9, java.io.IOException: Enable/Disable failed
Tue Jan 19 01:01:43 GMT-07:00 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@6515a0f9, java.io.IOException: Enable/Disable failed
Tue Jan 19 01:02:09 GMT-07:00 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@6515a0f9, java.io.IOException: Enable/Disable failed
Tue Jan 19 01:02:37 GMT-07:00 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@6515a0f9, java.io.IOException: Enable/Disable failed

编辑：

我在hbase-site.xml中没有主ip，我怎么才能找到它？不过，我有一份ips列表。我可以用spark-shell让它运行，但不能用spark-submit ..我收到这个错误，我觉得这在某种程度上与身份验证有关-

16/01/19 18:45:39 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Call to 10.115.201.64:60000 failed on local exception: java.io.IOException: Call id=4, waitTime=38 
org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Call to 10.115.201.64:60000 failed on local exception: java.io.IOException: Call id=4, waitTime=38 
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1607) 
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1633)
at ...

有什么建议吗？

Answer 1

你能分享更多关于你如何尝试连接的信息吗？我在这里附上一个例子，希望对你有用：

val hbaseOutConf = HBaseConfiguration.create()
hbaseOutConf.set("hbase.zookeeper.quorum", "list of ip's")
hbaseOutConf.set("hbase.zookeeper"+ ".property.clientPort","2181");
hbaseOutConf.set("hbase.master", "masterIP:60000");
hbaseOutConf.set("hadoop.security.authentication", "kerberos");
hbaseOutConf.set("hbase.security.authentication", "kerberos");

使用UserGRoupInformation从密钥表登录

UserGroupInformation.setConfiguration(hbaseOutConf);
UserGroupInformation.loginUserFromKeytab("user@---", keyTabPath);

HBaseAdmin.checkHBaseAvailable(hbaseOutConf);

如果您在Cloudera环境中，请检查以下项目：

http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/