Zendesk Maxwell撞车/坠落

Question

我们让maxwell运行在我们的产品中的docker上，它被配置为侦听MariadDb表上的更改，并将它们推送到kafka主题。我们的MariadDb大约有10-12个表，我们配置了maxwell，只听4个表，这4个表都被大量使用(写的次数太多了)。

最近我们看到我们的Maxwell正在下降，下面是我们在maxwell下降时发现的几个错误日志

Oct 28, 2022 @ 06:10:40.037 06:10:40,037 ERROR MysqlParserListener - (parse SET STATEMENT max_statement_time = 60 FOR flush table)

Oct 28, 2022 @ 06:10:40.040 06:10:40,040 ERROR SchemaChange - Error parsing SQL: 'SET STATEMENT max_statement_time=60 FOR flush table'

Oct 28, 2022 @ 06:10:40.047 06:10:40,045 ERROR AbstractSchemaStore - Error on bin log position Position[BinlogPosition[mysql-bin-changelog.049610:3209], lastHeartbeat=1666898518069]

正如最后一个错误日志显示的那样，在bin日志位置Position[BinlogPositionmysql-bin-changelog.049610:3209，上出现了错误，因此为了验证具有mysql-bin-changelog.049610:3209名称的bin日志文件是否存在，我们运行了SHOW binary logs;，并在上面命令输出的列表中找到了该文件。

请帮助我理解这里出了什么问题，并帮助解决问题。

环境详细信息

1. 操作系统: Centos
2. Platform - AWS实例(T4g介质)
3. CPU -2 CPU
4. 拉姆- 4GB

Maxwell版本- v1.37.7

备注：我们不处理DDL &我们的日志保留时间是3天

在Maxwell下降时记录的详细日志，很抱歉发布了大量日志:)

Oct 28, 2022 @ 06:10:39.084 06:10:39,082 INFO  ProducerConfig - ProducerConfig values: 
Oct 28, 2022 @ 06:10:39.146 06:10:39,146 INFO  AppInfoParser - Kafka commitId : aaa7af6d4a11b29d

Oct 28, 2022 @ 06:10:39.146 06:10:39,146 INFO  AppInfoParser - Kafka version : 1.0.0

Oct 28, 2022 @ 06:10:39.188 06:10:39,187 INFO  Maxwell - Maxwell v1.37.7 is booting (MaxwellKafkaProducer), starting at Position[BinlogPosition[mysql-bin-changelog.049610:3136], lastHeartbeat=1666898518069]

Oct 28, 2022 @ 06:10:39.377 06:10:39,377 INFO  MysqlSavedSchema - Restoring schema id 22 (last modified at Position[BinlogPosition[mysql-bin-changelog.046940:91534], lastHeartbeat=1666103424529])

Oct 28, 2022 @ 06:10:39.502 06:10:39,502 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at Position[BinlogPosition[mysql-bin-changelog.040524:418929], lastHeartbeat=0])

Oct 28, 2022 @ 06:10:39.555 06:10:39,555 INFO  MysqlSavedSchema - beginning to play deltas...

Oct 28, 2022 @ 06:10:39.576 06:10:39,576 INFO  MysqlSavedSchema - played 21 deltas in 15ms

Oct 28, 2022 @ 06:10:39.601 06:10:39,601 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: mysql-bin-changelog.049610:3136

Oct 28, 2022 @ 06:10:39.605 06:10:39,605 INFO  MaxwellHTTPServer - Maxwell http server starting

Oct 28, 2022 @ 06:10:39.607 06:10:39,607 INFO  MaxwellHTTPServer - Maxwell http server started on port 8080

Oct 28, 2022 @ 06:10:39.649 06:10:39,649 INFO  BinlogConnectorReplicator - Binlog connected.

Oct 28, 2022 @ 06:10:39.649 06:10:39,646 INFO  BinaryLogClient - Connected to mariadb-nlp.app.engati.local:3306 at mysql-bin-changelog.049610/3136 (sid:6379, cid:1560183)

Oct 28, 2022 @ 06:10:39.669 06:10:39,669 INFO  log - Logging initialized @2288ms to org.eclipse.jetty.util.log.Slf4jLog

Oct 28, 2022 @ 06:10:39.902 06:10:39,902 INFO  Server - jetty-9.4.41.v20210516; built: 2021-05-16T23:56:28.993Z; git: 98607f93c7833e7dc59489b13f3cb0a114fb9f4c; jvm 11.0.15+10

Oct 28, 2022 @ 06:10:40.037 06:10:40,037 ERROR MysqlParserListener - (parse SET STATEMENT max_statement_time = 60 FOR flush table)

Oct 28, 2022 @ 06:10:40.040 06:10:40,040 ERROR SchemaChange - Error parsing SQL: 'SET STATEMENT max_statement_time=60 FOR flush table'

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.Maxwell.main(Maxwell.java:336)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.Maxwell.start(Maxwell.java:227)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:301)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:228)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:723)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:400)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:378)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:102)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:49)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:108)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94)

Oct 28, 2022 @ 06:10:40.047     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)

Oct 28, 2022 @ 06:10:40.047     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17)

Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93)

Oct 28, 2022 @ 06:10:40.047 com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SET

Oct 28, 2022 @ 06:10:40.047 06:10:40,045 ERROR AbstractSchemaStore - Error on bin log position Position[BinlogPosition[mysql-bin-changelog.049610:3209], lastHeartbeat=1666898518069]

Oct 28, 2022 @ 06:10:40.047 06:10:40,044 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@5628733f{/,null,AVAILABLE}

Oct 28, 2022 @ 06:10:40.075 06:10:40,074 INFO  Server - Started @2700ms

Oct 28, 2022 @ 06:10:40.075 06:10:40,074 INFO  AbstractConnector - Started ServerConnector@7152d3c1{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}

Oct 28, 2022 @ 06:10:40.129 06:10:40,129 INFO  BinlogConnectorReplicator - Binlog disconnected.

Oct 28, 2022 @ 06:10:40.149 06:10:40,149 INFO  TaskManager - Stopping 5 tasks

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.8-1.jar:4.8-1]

Oct 28, 2022 @ 06:10:40.187     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17) ~[antlr4-runtime-4.8-1.jar:4.8-1]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187 com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SET

Oct 28, 2022 @ 06:10:40.187 06:10:40,152 ERROR TaskManager - cause: 

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.Maxwell.main(Maxwell.java:336) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.Maxwell.start(Maxwell.java:227) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:301) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:228) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:723) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:400) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:378) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:102) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:49) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:108) ~[maxwell-1.37.7.jar:1.37.7]

Oct 28, 2022 @ 06:10:40.189 06:10:40,189 INFO  TaskManager - Stopping: com.zendesk.maxwell.schema.PositionStoreThread@51b3ee1b

Oct 28, 2022 @ 06:10:40.191 06:10:40,189 INFO  TaskManager - Stopping: com.zendesk.maxwell.producer.MaxwellKafkaProducerWorker@5ec6ffb6

Oct 28, 2022 @ 06:10:40.192 06:10:40,192 INFO  KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

Oct 28, 2022 @ 06:10:40.201 06:10:40,201 INFO  TaskManager - Stopping: com.zendesk.maxwell.monitoring.MaxwellHTTPServerWorker@1490968e

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.Maxwell.main(Maxwell.java:336)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.Maxwell.start(Maxwell.java:227)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:301)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:228)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:723)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:400)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:378)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:102)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:49)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:108)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94)

Oct 28, 2022 @ 06:10:40.201     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)

Oct 28, 2022 @ 06:10:40.201     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17)

Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93)

Oct 28, 2022 @ 06:10:40.201 com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SET

Oct 28, 2022 @ 06:10:40.201 06:10:40,199 INFO  TaskManager - Stopping: com.zendesk.maxwell.replication.BinlogConnectorReplicator@5a1d68e0

Oct 28, 2022 @ 06:10:40.201 06:10:40,198 INFO  TaskManager - Stopping: com.zendesk.maxwell.bootstrap.BootstrapController@3857c48d

Oct 28, 2022 @ 06:10:40.220 06:10:40,219 INFO  AbstractConnector - Stopped ServerConnector@7152d3c1{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}

Oct 28, 2022 @ 06:10:40.223 06:10:40,223 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@5628733f{/,null,STOPPED}

ct 28, 2022 @ 06:10:41.637  06:10:41,636 INFO  TaskManager - Stopped all tasks

Answer 1

java回溯显示这是Maxwell中的一个错误。这个被报道为1931年问题。

问题是，v1.37.7中的Maxwell并没有针对类似于SET STATEMENT ... FOR的黑名单解析FLUSH，这意味着它不采取任何行动。

由于上游的响应能力很强，所以现在已经在1.39.2版中对此进行了修正。