卡夫卡偏移滞后的原因是什么？

Question

我们一直在尝试修复不想使用数据的消费者。它每4秒轮询一次，通常只收到大约100条记录。最大拉力约为18000。然后，它将原始字节写入文件。都是快速操作。

但是消费者组呼叫显示如下

[xyz@AZ254PP01 kafka]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --new-consumer --group postproc-red2
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
postproc-red2, targeting-seg, 0, 21921226, 21921388, 162, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 0, 72891022, 78349037, 5458015, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 0, 16665679, 18122272, 1456593, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 0, 50847858, 56468186, 5620328, consumer-2_/192.168.201.121
postproc-red2, targeting-seg, 1, 20733636, 24858954, 4125318, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 1, 75084847, 89556283, 14471436, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 1, 19648100, 19648100, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 1, 66549015, 66549015, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-seg, 2, 24903106, 28041684, 3138578, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 2, 99480656, 99480656, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 2, 19982077, 24848188, 4866111, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 2, 69611742, 76253273, 6641531, consumer-2_/192.168.201.121

当它记录日志时，它似乎只想使用targeting seg-0，但一次只消耗60-90条记录。它有时会一次从所有0个分区中提取几千个。这些数据没有任何意义，当谈到在哪里寻找问题时，我们不知所措。它只有一个消费者。一个使用这些主题的所有分区的进程。

最近两天的生产者似乎只写入了0个分区。我们也不知道其中的原因。

我试着查看它是否可以在所有超过60秒未被拉出的分区上运行seekToEnd。

14:58:33.896 [Kafka-0] INFO  c.t.e.p.s.red2.ConsumerService - Processing records 3263 of Topics [targeting-seg-0, targeting-visit-0, targeting-vis-0, targeting-vc-0]
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vc-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vc-2
14:58:38.005 [Kafka-0] INFO  c.t.e.p.s.red2.ConsumerService - Processing records 3284 of Topics [targeting-seg-0, targeting-visit-0, targeting-vis-0, targeting-vc-0]
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-2
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-2
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-2

代码是订阅(主题)。民意测验(500)。seekToEnd。然后，我希望在下一次投票中，seekToEnd会修复一个有问题的偏移量。

任何想法的问题可能是卡夫卡设置或如何解决它。它基本上是在默认设置下运行的。所有分区都没有副本->复制因子1。我们打算切换到复制因子3，因为生产者是三个服务器。

Answer 1

生产者根据将要“产生”的消息的关键字的散列来选择目标分区。您应该检查最近两天的消息是否总是具有相同的密钥。

在消费者端，你确定postproc-red2只用于这个消费者吗？您确定在同一群内有唯一的消费者id吗？Hovewer的最佳实践是让使用者数量等于源主题的分区数量。

希望这能有所帮助