卡夫卡连接分布式NOT_ENOUGH_REPLICAS错误

Question

我正在设置kafka s3接收器连接器，当我尝试使用connect-standalone.sh时，一切正常。

现在，我切换到connect-distributed.sh，遇到了以下错误：

WARN [Producer clientId=producer-3] Got error produce response with correlation id 40 on topic-partition connect-configs-0, retrying (2147483611 attempts le
ft). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:637)

我试着改变

offset.storage.replication.factor=5
config.storage.replication.factor=5
status.storage.replication.factor=5

这是副本的实际数量，但是这个错误并没有消失。

我还试图更改所有的值以匹配min.insync.replicas，但它仍然不起作用。

Topic: myTopic    PartitionCount: 12      ReplicationFactor: 5    Configs: compression.type=producer,***min.insync.replicas=2***,segment.bytes=1073741824,max.me
ssage.bytes=2097152,index.interval.bytes=4096,retention.bytes=-1,segment.index.bytes=10485760
        Topic: myTopic        Partition: 0    Leader: 43      Replicas: 43,11,9,42,10 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 1    Leader: 11      Replicas: 11,42,10,43,9 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 2    Leader: 9       Replicas: 9,42,10,43,11 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 3    Leader: 42      Replicas: 42,10,43,11,9 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 4    Leader: 10      Replicas: 10,43,11,9,42 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 5    Leader: 43      Replicas: 43,10,11,9,42 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 6    Leader: 11      Replicas: 11,43,9,42,10 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 7    Leader: 9       Replicas: 9,43,11,42,10 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 8    Leader: 42      Replicas: 42,11,9,10,43 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 9    Leader: 10      Replicas: 10,42,43,11,9 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 10   Leader: 43      Replicas: 43,11,9,42,10 Isr: 10,42,9,11,43
        Topic: myTopic        Partition: 11   Leader: 11      Replicas: 11,42,10,43,9 Isr: 10,42,9,11,43


root@9c4c4d97dcd6:/opt/bitnami/kafka/bin# grep replication.factor= /plugins/worker.properties
offset.storage.replication.factor=2
config.storage.replication.factor=2
status.storage.replication.factor=2

我有什么不正确的配置吗？谢谢

Answer 1

看来，我只需要在kafka中重新创建以下主题：

offset.storage.topic=connect-offsets-distriuted
config.storage.topic=connect-configs-distributed
status.storage.topic=connect-status-distributed

请注意，我添加了-distributed后缀。我以前

offset.storage.topic=connect-offsets
config.storage.topic=connect-configs
status.storage.topic=connect-status

这些遗留下来的主题不适用于连接器分发。

此外，它还需要与服务器端的复制因子匹配。

Topic: connect-status-distributed       PartitionCount: 5       ReplicationFactor: 2    Configs: compression.type=producer,min.insync.replicas=2,cleanup.policy=compact,segment.bytes=10
73741824,max.message.bytes=2097152,index.interval.bytes=4096,retention.bytes=-1,segment.index.bytes=10485760
        Topic: connect-status-distributed       Partition: 0    Leader: 43      Replicas: 43,11 Isr: 43,11
        Topic: connect-status-distributed       Partition: 1    Leader: 11      Replicas: 11,42 Isr: 11,42
        Topic: connect-status-distributed       Partition: 2    Leader: 9       Replicas: 9,42  Isr: 9,42
        Topic: connect-status-distributed       Partition: 3    Leader: 42      Replicas: 42,10 Isr: 42,10
        Topic: connect-status-distributed       Partition: 4    Leader: 10      Replicas: 10,43 Isr: 10,43
Topic: connect-configs-distributed      PartitionCount: 1       ReplicationFactor: 2    Configs: compression.type=producer,min.insync.replicas=2,cleanup.policy=compact,segment.bytes=10
73741824,max.message.bytes=2097152,index.interval.bytes=4096,retention.bytes=-1,segment.index.bytes=10485760
        Topic: connect-configs-distributed      Partition: 0    Leader: 11      Replicas: 11,43 Isr: 11,43