Artemis日志失败导致artemis故障转移

Question

我已经运行了Artemis副本集与主人和两个奴隶和有问题的artemis节点。它不时地失败。下面是失败的artemis节点的日志：

2021-04-14 02:11:15,997 WARN  [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ed72f65c-9cb5-11eb-8b65-001dd8b71e33
2021-04-14 02:11:15,997 WARN  [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ed72f65c-9cb5-11eb-8b65-001dd8b71e33
2021-04-14 02:11:16,139 WARN  [org.apache.activemq.artemis.journal] "Reference Handler" daemon prio=10 Id=2 RUNNABLE
    at java.base@14.0.1/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
    at java.base@14.0.1/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
    at java.base@14.0.1/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
2021-04-14 02:11:16,140 WARN  [org.apache.activemq.artemis.journal] "Finalizer" daemon prio=8 Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@47d5008c
    at java.base@14.0.1/java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.ReferenceQueue$Lock@47d5008c
    at java.base@14.0.1/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
    at java.base@14.0.1/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
    at java.base@14.0.1/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
2021-04-14 02:11:16,140 WARN  [org.apache.activemq.artemis.journal] "Signal Dispatcher" daemon prio=9 Id=4 RUNNABLE
2021-04-14 02:11:16,140 WARN  [org.apache.activemq.artemis.journal] "Notification Thread" daemon prio=9 Id=9 RUNNABLE
2021-04-14 02:11:16,140 WARN  [org.apache.activemq.artemis.journal] "Common-Cleaner" daemon prio=8 Id=10 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@4912e606
    at java.base@14.0.1/java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.ReferenceQueue$Lock@4912e606
    at java.base@14.0.1/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
    at java.base@14.0.1/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
    at java.base@14.0.1/java.lang.Thread.run(Thread.java:832)
    at java.base@14.0.1/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:134)
2021-04-14 02:11:16,140 WARN  [org.apache.activemq.artemis.journal] "ActiveMQ Artemis Server Shutdown Timer" daemon prio=5 Id=12 TIMED_WAITING on java.util.TaskQueue@4d2d6bfc
    at java.base@14.0.1/java.lang.Object.wait(Native Method)
    -  waiting on java.util.TaskQueue@4d2d6bfc
    at java.base@14.0.1/java.util.TimerThread.mainLoop(Timer.java:553)
    at java.base@14.0.1/java.util.TimerThread.run(Timer.java:506)
2021-04-14 02:11:16,140 WARN  [org.apache.activemq.artemis.journal] "Thread-0 (-scheduled-threads)" prio=5 Id=18 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@82fec3d
    at java.base@14.0.1/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@82fec3d
    at java.base@14.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
    at java.base@14.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1661)
    at java.base@14.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
    at java.base@14.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
    at java.base@14.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1056)
    at java.base@14.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1116)
    at java.base@14.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
...
2021-04-14 02:11:16,146 WARN  [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/home/artemis-broker/data/journal/activemq-data-3.amq, message=Timeout on close: java.io.IOException: Timeout on close
    at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.close(AIOSequentialFile.java:127) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.close(AIOSequentialFile.java:104) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.closeFile(JournalFilesRepository.java:534) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.moveNextFile(JournalImpl.java:3214) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.switchFileIfNecessary(JournalImpl.java:3153) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendRecord(JournalImpl.java:2883) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.access$100(JournalImpl.java:92) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl$1.run(JournalImpl.java:850) [artemis-journal-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.16.0.jar:2.16.0]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.16.0.jar:2.16.0]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [java.base:]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [java.base:]
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.16.0.jar:2.16.0]

2021-04-14 02:11:16,151 WARN  [org.apache.activemq.artemis.journal] waiting pending callbacks on activemq-data-3.amq from 10 seconds!
...

但我真的不明白这个问题有什么问题？我没有足够的尺寸来写日记或其他什么的吗？

Answer 1

这是日志中最重要的部分：

2021-04-14 02:11:16,146 WARN  [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/home/artemis-broker/data/journal/activemq-data-3.amq, message=Timeout on close: java.io.IOException: Timeout on close

这是一个“关键的IO错误”，因此代理将关闭自己。您应该迁移到ActiveMQ Artemis2.17.0，因为本期已经解决了。