如何从Apache Cassandra CommitLogReplayException恢复

Question

在运行apache-cassandra 3.3时，我意外地关闭了Mac。在新的bootload中重启cassandra时，我得到了以下错误：

INFO  15:25:15 Replaying /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351787.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461942807785.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461943004975.log
ERROR 15:25:15 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 542295 of commit log /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, with bad position but valid CRC
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:105) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:78) [apache-cassandra-3.5.jar:3.5]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) [guava-18.0.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) [guava-18.0.jar:na]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:399) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.jar:3.5]

我在stack overflow和google上搜索，没有找到同样问题的帖子。我发现了相关的问题https://issues.apache.org/jira/browse/CASSANDRA-9749，但这个问题被标记为已解决，并且与检测到问题后继续执行有关，而不是由于异常而失败。

在这种情况下我该怎么办？是否有任何恢复路径(假设没有可用的快照)？有没有办法截断违规记录中的提交日志？有没有工具可以修复坏的标题(例如删除记录)？

谢谢。

Answer 1

当我在Mac OS上玩我本地的Cassandra时，有时也会遇到这个例外。

如果您不关心丢失提交日志中的数据，或者如果您在某个地方有备份，只需停止Cassandra，清理commitlog文件夹，然后重新启动。

恐怕别无选择。顺便问一下，您是否在Mac OS上部署Cassandra用于生产？

• 如果是，您应该知道

OS不受官方支持

• 如果它不是用于生产，那么谁会关心它，只需清理commitlog并重启

Answer 2

在Windows7(本地机器)环境下运行Cassandra时，我遇到了类似的问题。我在日志中发现的异常是

错误07:14:35初始化期间处理提交日志时出错，导致退出。org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:无法读取位于org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:616) apache-cassandra-3.7.0.jar:3.7.0 at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:378) apache-cassandra-3.7.0.jar:3.7.0的C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1470969948498.log文件中的提交日志描述符

由于这只是一个测试设置，我删除了特定的提交日志文件，并从windows服务管理器重新启动它，它完全启动了。如果它发生在生产环境中，我不确定该怎么办。

Answer 3

第二次我在我的单节点开发系统上得到了这个。如果jvm意外终止，似乎会发生这种情况。清除提交日志目录解决了这个问题。使用Windows 10。

让我担心的是，即使没有对数据库进行更新，我也会遇到这些错误问题。卡珊德拉不是经常冲厕所吗。我不希望仅仅因为服务器崩溃或类似的原因就在生产中出现类似的问题，即使我有副本节点。

在stdout日志中，最后一部分是

INFO  06:17:39 Replaying C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812251.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471411951134.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471454506802.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471532812678.log
ERROR 06:17:39 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:373) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.4.0.jar:3.4.0]