凤凰连接在运行一段时间后抛出错误消息“未能找到任何Kerberos tgt”。

Question

我正在使用凤凰连接到安全的HBase。当客户端启动时，它运行良好。我可以连接到HBase，从HBase查询数据。菲尼克斯JDBC连接位于连接池中，不关闭。但几个小时后，我无法使用相同的连接从HBase查询任何内容。为了使我的应用程序工作，我必须重新启动客户机。错误消息是：

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Fri Nov 11 06:24:01 CST 2016, RpcRetryingCaller{globalStartTime=1478867041301, pause=100, retries=1}, java.io.IOException: Could not set up IO Streams to <regionserver>/<ip_address>:60020
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getTable(MetaDataProtos.java:11769) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1301) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1288) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
    at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1737) ~[hbase-client-1.0.1.jar:1.0.1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
    ... 1 common frames omitted
Caused by: java.io.IOException: Could not set up IO Streams to <regionserver>/<ip_address>:60020
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:772) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:880) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:849) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1173) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:31913) ~[hbase-protocol-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1605) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ~[hbase-client-1.0.1.jar:1.0.1]
    ... 10 common frames omitted
Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:672) ~[hbase-client-1.0.1.jar:1.0.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
    at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_79]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:630) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:738) ~[hbase-client-1.0.1.jar:1.0.1]
    ... 20 common frames omitted
    Caused by: javax.security.sasl.SaslException: GSS initiate failed
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) ~[na:1.7.0_79]
    at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:604) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:153) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:730) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:727) ~[hbase-client-1.0.1.jar:1.0.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
    at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_79]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:727) ~[hbase-client-1.0.1.jar:1.0.1]
    ... 20 common frames omitted
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_79]
    at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_79]
    at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_79]
    at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_79]
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_79]
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_79]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[na:1.7.0_79]
    ... 29 common frames omitted

Answer 1

凭据可能已经过期。在以下情况下，在客户端和区域服务器执行klist以查看有效的票证：

• 你可以连接
• 你不能连接

Answer 2

我们的env: HDP 2.5

当它即将到期时，我们应该按照它来进行自动更新。

一天后，Kerberos票就过期了。因此，在重新启动时，它将工作，并持续24小时。我们可以通过这种方式自动更新HBase Kerberos connection renewal strategy