Cassandra驱动程序异常“所有尝试查询失败的主机”每隔几个小时发生一次，而不作解释。

Question

我的Cassandra集群(4个节点集群)有问题。Cassandra版本为2.2.9，驱动程序版本为3.0.3。

几个小时后(~ 3个小时)，我在驱动程序日志中看到以下问题：

1. OutOfDirectMemoryError (偶发性的，大多数情况下不受影响)
2. 没有协议版本匹配整数版本
3. 未知响应操作码
4. 心跳查询超时
5. 所有尝试查询的主机都失败了->不能再查询Cassandra了

Cassandra集群是健康的，当我重新启动应用程序时，一切都会工作几个小时。

日志片段：

First Time                       Count  Message
2017-11-11 19:03:03 +0100            51  [/??.???.??.??:????] preparing to open ? new connections, total = ???
2017-11-11 19:03:03 +0100            49  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=?, closed=false] Transport initialized, connection ready
2017-11-11 19:03:03 +0100            24  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=?, closed=true] closed, remaining = ???
2017-11-11 19:03:29 +0100             1  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate ??????? byte(s) of direct memory (used: ???????, max: ????????))
2017-11-11 19:03:29 +0100            14  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=???, closed=false] failed, remaining = ???
2017-11-11 19:03:29 +0100             7  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=??, closed=false] failed, remaining = ???
2017-11-11 19:03:29 +0100             1  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: No protocol version matching integer version ?)
2017-11-11 19:03:29 +0100             5  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ??)
2017-11-11 19:03:29 +0100             4  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ?)
2017-11-11 19:03:29 +0100             3  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode -???)
2017-11-11 19:03:30 +0100             3  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ??)
2017-11-11 19:03:30 +0100             2  Defuncting Connection[/??.???.??.??:????-???, inFlight=?, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ??)
2017-11-11 19:03:30 +0100           401  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=?, closed=false] failed, remaining = ???
2017-11-11 19:03:33 +0100             1  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ???)
2017-11-11 19:03:41 +0100           722  Defuncting Connection[/??.???.??.??:????-???, inFlight=?, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:41 +0100             8  [/??.???.??.??:????] Connection[/??.???.??.??:????-?, inFlight=?, closed=false] failed, remaining = ???
2017-11-11 19:03:41 +0100            11  Defuncting Connection[/??.???.??.??:????-?, inFlight=?, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:41 +0100            67  [/??.???.??.??:????] Connection[/??.???.??.??:????-??, inFlight=?, closed=false] failed, remaining = ???
2017-11-11 19:03:41 +0100           115  Defuncting Connection[/??.???.??.??:????-??, inFlight=?, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:44 +0100             2  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ?)
2017-11-11 19:03:51 +0100             2  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:51 +0100           265  Failed to post timeseries data Error Returned - 
2017-11-11 19:03:57 +0100             3  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:04:01 +0100            39  Defuncting Connection[/??.???.??.??:????-???, inFlight=?, closed=false] because: [/??.???.??.??:????] Operation timed out
2017-11-11 19:04:01 +0100            12  Error processing jobs: execution of statement failed:All host(s) tried for query failed (tried: /??.???.??.??:???? (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /??.???.??.??:???? (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /??.???.??.??:???? (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /??.???.??.??:???? [only showing errors of first ? hosts, use getErrors() for more details])

有人知道什么是根本原因吗？

Answer 1

请检查应用程序使用的直接内存设置。确保有足够的可用内存，可以由驱动程序分配。卡桑德拉需要分配直接内存。在无法分配内存的情况下，我已经看到类似的问题，即使它与内存相关，也被报告为NoHostAvailableException。

Answer 2

Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate ??????? byte(s) of direct memory (used: ???????, max: ????????))

你的记忆有点问题。只要存在这些问题，您就不能期望驱动程序正常工作。您还说您的应用程序在几个小时后停止工作。在我看来，您的应用程序中似乎有内存泄漏。