Cassandra 4- java.lang.OutOfMemoryError:无法创建新的原生线程

Question

在将Cassandra 3.x升级到4.X之后，会发生错误。

INFO  [NonPeriodicTasks:1] 2022-09-24T04:41:26,003 SSTable.java:111 - Deleting sstable: XXX/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/nb-152737-big
INFO  [COMMIT-LOG-ALLOCATOR] 2022-09-24T04:44:49,023 ColumnFamilyStore.java:878 - Enqueuing flush of in_production_checks: 21.660KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [COMMIT-LOG-ALLOCATOR] 2022-09-24T04:44:49,024 ColumnFamilyStore.java:878 - Enqueuing flush of ingestion_hosts_v3: 323.955KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [COMMIT-LOG-ALLOCATOR] 2022-09-24T04:44:49,025 ColumnFamilyStore.java:878 - Enqueuing flush of spock_cluster_leases: 2.563KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:940] 2022-09-24T04:44:49,081 Memtable.java:471 - Writing Memtable-in_production_checks@1168154357(6.122KiB serialized bytes, 25 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
INFO  [PerDiskMemtableFlushWriter_0:940] 2022-09-24T04:44:49,082 Memtable.java:500 - Completed flushing XXX/cassandra/data/keyspace1/in_production_checks-7810e5c0922511ea9d87c1aca011cf3c/nb-182421-big-Data.db (3.813KiB) for commitlog position CommitLogPosition(segmentId=1663918207131, position=25214)
INFO  [PerDiskMemtableFlushWriter_0:941] 2022-09-24T04:44:49,086 Memtable.java:471 - Writing Memtable-ingestion_hosts_v3@1316569071(58.113KiB serialized bytes, 5831 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
INFO  [PerDiskMemtableFlushWriter_0:941] 2022-09-24T04:44:49,091 Memtable.java:500 - Completed flushing XXX/cassandra/data/keyspace1/ingestion_hosts_v3-762f7b90922511ea9d87c1aca011cf3c/nb-399812-big-Data.db (44.660KiB) for commitlog position CommitLogPosition(segmentId=1663918207131, position=25214)
INFO  [PerDiskMemtableFlushWriter_0:940] 2022-09-24T04:44:49,218 Memtable.java:471 - Writing Memtable-spock_cluster_leases@1265068081(0.164KiB serialized bytes, 78 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
INFO  [PerDiskMemtableFlushWriter_0:940] 2022-09-24T04:44:49,218 Memtable.java:500 - Completed flushing XXX/cassandra/data/keyspace2/tbl1-78b6e9c0922511ea9d87c1aca011cf3c/nb-278783-big-Data.db (0.135KiB) for commitlog position CommitLogPosition(segmentId=1663918207131, position=25214)
ERROR [GossipTasks:1] 2022-09-24T04:45:11,989 DefaultPromise.java:844 - Failed to submit a listener notification task. Event loop shut down?
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) ~[?:1.8.0_345]
        at java.lang.Thread.start(Thread.java:719) ~[?:1.8.0_345]
        at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:232) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:208) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:842) [netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenerWithStackOverFlowProtection(DefaultPromise.java:529) [netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListener(DefaultPromise.java:477) [netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.CompleteFuture.addListener(CompleteFuture.java:48) [netty-all-4.1.58.Final.jar:4.1.58.Final]
        at org.apache.cassandra.net.FutureCombiner.(FutureCombiner.java:83) [apache-cassandra-4.0.5.jar:4.0.5]
        at org.apache.cassandra.net.FutureCombiner.(FutureCombiner.java:55) [apache-cassandra-4.0.5.jar:4.0.5]
        at org.apache.cassandra.net.InboundSockets.open(InboundSockets.java:235) [apache-cassandra-4.0.5.jar:4.0.5]
        at org.apache.cassandra.net.MessagingService.waitUntilListening(MessagingService.java:534) [apache-cassandra-4.0.5.jar:4.0.5]
        at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:287) [apache-cassandra-4.0.5.jar:4.0.5]
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) [apache-cassandra-4.0.5.jar:4.0.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_345]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_345]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_345]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_345]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_345]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_345]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.58.Final.jar:4.1.58.Final]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_345]

我使用的是旧的java客户机(3.9.0)和Cassandra 4.x。错误的频率不经常。通常情况下，这种情况很少发生在6小时以上。

设置信息。

# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63939
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63939
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

卡桑德拉正在和-Xss256k -Xms4020M -Xmx4020M -Xmn1005M一起跑。在卡桑德拉3.x号上，一切都很好。

# java -version
openjdk version "1.8.0_345"

Answer 1

在您提供的信息中，除了JVM内存不足之外，没有任何明显的问题指向其他问题。当创建了许多线程时，就会发生这种情况。

您将需要使用像jstack这样的工具进行线程转储，并对输出进行分析以寻找线索。你要寻找的东西是：

• 正在创建哪些线程？
• 顶部线程(按计数)

如果您以前没有做过线程转储分析，那么您需要做一些研究来了解所需的内容。请注意，分析更多的是一门艺术，而不是科学。

一个临时的解决方法是，对于非常低的生产工作负载，将堆增加到至少8GB。干杯!