Kubernetes DSE Cassandra CommitLogReplayer$CommitLogReplayException

Question

我在Kubernetes (9个吊舱)上安装了卡桑德拉，除了一个吊舱外，所有的吊舱都已经启动并运行，这显示了下面的错误。

org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.shouldSkipSegmentOnError(CommitLogReplayer.java:438)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleUnrecoverableError(CommitLogReplayer.java:452)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:109)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:84)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:236)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:134)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:154)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:213)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:194)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)
    at com.datastax.bdp.DseModule.main(DseModule.java:96)
Caused by: org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:111)
    ... 12 more
ERROR [main] 2021-09-06 06:19:08,990  JVMStabilityInspector.java:251 - JVM state determined to be unstable.  Exiting forcefully due to:
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.shouldSkipSegmentOnError(CommitLogReplayer.java:438)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleUnrecoverableError(CommitLogReplayer.java:452)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:109)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:84)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:236)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:134)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:154)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:213)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:194)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)
    at com.datastax.bdp.DseModule.main(DseModule.java:96)

有人能帮帮我吗？

Answer 1

由于任何原因，提交日志段中的一个在节点上损坏。

您可以通过手动删除pod上的文件来解决这个问题：

/var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log

有趣的是，提交日志段是在9月2日(1630582314923)创建的，但是您发布的日志条目是从9月6日开始的。

您需要检查荚上的Cassandra日志(而不是pod日志本身)，以确定根本原因并解决它。干杯!