修复过程中的Cassandra异常

Question

我目前在运行nodetool修复时遇到了cassandra的问题。

我在每个cassandra节点上运行了一次nodetool修复，遇到了下面提到的异常。

在做了一些阅读之后，我运行了"nodetool scrub“和"sstablescrub”，但没有成功，我仍然在运行nodetool修复时看到相同的错误。

我们使用的是cassandra版本2.0.1。

还有没有人见过这个问题？

ERROR [AntiEntropySessions:3] 2013-11-26 15:25:03,315 RepairSession.java (line 278) [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c] session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]] Validation failed in /<ipaddress-1>
        at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
        at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
ERROR [AntiEntropySessions:3] 2013-11-26 15:25:03,315 CassandraDaemon.java (line 185) Exception in thread Thread[AntiEntropySessions:3,5,RMI Runtime]
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]] Validation failed in /<ipaddress-1>
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]] Validation failed in /<ipaddress-1>
        at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
        at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        ... 3 more
 INFO [AntiEntropySessions:4] 2013-11-26 15:25:03,319 RepairSession.java (line 236) [repair #fa075b20-56f1-11e3-9885-5938b4e97c9c] new session: will sync /<ipaddress1>, /<ipaddress2>, /<ipaddress3> on range (2637939872511762724,2642446772090452246] for <keyspace>.[<table-1>, <table-2>, <table-3>, <table-4>, <table-5>]
 INFO [AntiEntropySessions:4] 2013-11-26 15:25:03,319 RepairJob.java (line 116) [repair #fa075b20-56f1-11e3-9885-5938b4e97c9c] requesting merkle trees for <table-2> (to [/<ipaddress1>, <ipaddress2>, <ipaddress3>])
ERROR [ValidationExecutor:5] 2013-11-26 15:25:03,353 Validator.java (line 242) Failed creating a merkle tree for [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]], /<ipaddress-2> (see log for details)
ERROR [ValidationExecutor:5] 2013-11-26 15:25:03,353 CassandraDaemon.java (line 185) Exception in thread Thread[ValidationExecutor:5,1,main]
java.lang.AssertionError
        at org.apache.cassandra.db.compaction.PrecompactedRow.update(PrecompactedRow.java:171)
        at org.apache.cassandra.repair.Validator.rowHash(Validator.java:198)
        at org.apache.cassandra.repair.Validator.add(Validator.java:151)
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:798)
        at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:60)
        at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:395)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Answer 1

根据您的数据模型，您可能会看到CASSANDRA-6152的影响。有关详细信息，请参阅此评论：

https://issues.apache.org/jira/browse/CASSANDRA-6152?focusedCommentId=13793977&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13793977

无论如何，你应该立即更新到2.0.3，因为在你的当前版本中还有其他几个相当严重的bug。