datastax -com.datastax.shaded.netty.channel.ConnectTimeoutException中的意外连接超时

Question

在应用程序第一次启动时，我们就在会话对象中获得连接。我们的调度程序一直用它把数据插入卡桑德拉。但在某个时间点，它的连接超时，然后它与Cassandra和应用程序重新连接正常工作。

调度程序的频繁超时。(我把我的ip改为.在这个职位上)

2015-08-19 09:19:36 DEBUG Connection:274 - Defuncting connection to /*.*.*.*:9042
com.datastax.driver.core.TransportException: [/10.0.*.*:9042] Cannot connect
    at com.datastax.driver.core.Connection.<init>(Connection.java:104)
    at com.datastax.driver.core.Connection$Factory.open(Connection.java:544)
    at com.datastax.driver.core.Cluster$Manager$5.tryReconnect(Cluster.java:1652)
    at com.datastax.driver.core.AbstractReconnectionHandler.run(AbstractReconnectionHandler.java:124)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
    at com.datastax.shaded.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at com.datastax.shaded.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    ... 3 more
2015-08-19 09:19:36 DEBUG Connection:449 - Connection[/*.*.*.*:9042-8, inFlight=0, closed=true] closing connection
2015-08-19 09:19:36 DEBUG Connection:721 - Connection[/*.*.*.*:9042-8, inFlight=0, closed=false] connection error
com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
    at com.datastax.shaded.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at com.datastax.shaded.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
2015-08-19 09:19:36 DEBUG Cluster:1681 - Failed reconnection to /10.0.1.18:9042 ([/*.*.*.*:9042] Cannot connect), scheduling retry in 128000 milliseconds
2015-08-19 09:19:36 DEBUG Connection:274 - Defuncting connection to /*.*.*.*:9042
com.datastax.driver.core.TransportException: [/*.*.*.*:9042] Unexpected exception triggered (com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042)
    at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:727)
    at com.datastax.shaded.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at com.datastax.shaded.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at com.datastax.shaded.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
    at com.datastax.shaded.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at com.datastax.shaded.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:141)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
    at com.datastax.shaded.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at com.datastax.shaded.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
    ... 8 more

Answer 1

看起来你在这里碰到了连接超时。一些关于可能发生的事情的想法。

1. 您在SocketOptions中的连接超时调优过低。您是否显式地配置它？如果不是，缺省值是5000 is。
2. 您在cassandra节点的rpc_address上的cassandra.yaml配置为您的客户端无法访问的私有IP接口。我注意到日志(Failed reconnection to /10.0.1.18:9042)中存在一个10.x.x.xIP。驱动程序使用rpc_address (或者broadcast_rpc_address (如果rpc_address为0.0.0.0)连接到cassandra主机。这有可能吗？
3. 您的C*节点重载。如果发生这种情况，我怀疑这是C*节点，因为通常情况下，您会得到操作(读取)超时，因为承载cassandra的操作系统接受连接，但是您的C* java进程不会响应请求，因为它正经历频繁垃圾收集的高暂停率。在任何情况下，我都会检查cassandra节点上的system.log，看看是否有任何指示主机上下运行、暂停时间较长的GCInspector或其他可能表示问题的消息。