为什么filebeat在把数据插入到kafka的话题中时会注册一个新的经纪人？

Question

我尝试使用file节拍将数据插入到cta_test主题中。

然而，cta_test的主题没有数据。

所以我检查了文件记录，在那里找到了一个新的代理注册部分。

我打算从filebeat.yml文件访问kafka到域"kafka-001-cluster.priv.com ~ kafka-003-cluster.priv.com“。

但连接被重置为“Kafka-001.public.com~Kafka-003.public.com”。

我想知道为什么它被重置为kafka-00*.public.com，而不是我想要的kafka-00*-cluster.priv.com。

• 文件传输版本: 6.3.0
• kafka版本:2.0.0
• 文件击败日志

## first try log ##
2020-12-30T09:55:07.734+0900    DEBUG   [kafka] kafka/client.go:69      connect: [kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092 kafka-003-cluster.priv.com:9092]
2020-12-30T09:55:07.734+0900    INFO    kafka/log.go:36 kafka message: Initializing new client
2020-12-30T09:55:07.734+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for all topics from broker kafka-003-cluster.priv.com:9092
2020-12-30T09:55:07.734+0900    DEBUG   [harvester]     log/log.go:85   End of file reached: /home/logs/access_log; Backoff now.
2020-12-30T09:55:07.794+0900    INFO    kafka/log.go:36 Connected to broker at kafka-003-cluster.priv.com:9092 (unregistered)
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 client/brokers registered new broker #2 at kafka-002.publ.com:9092
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 client/brokers registered new broker #1 at kafka-001.publ.com:9092
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 client/brokers registered new broker #3 at kafka-003.publ.com:9092
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 kafka message: Successfully initialized new client
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 producer/broker/3 starting up
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9

2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/broker/3 state ctange to [closing] because dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state ctange to [retrying-1]
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/broker/3 shut down


2020-12-30T09:55:37.895+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.cluster.priv.com:9092
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/broker/3 starting up
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 selected broker 3
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state ctange to [flushing-1]
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state ctange to [normal]


## retry log##
kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/broker/3 state change to [closing] because dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 state change to [retrying-1]
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 abandoning broker 3
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state change to [retrying-2]
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/broker/3 shut down
2020-12-30T15:13:41.940+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092
2020-12-30T15:13:41.940+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/broker/3 starting up
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/9
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 selected broker 3
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state change to [flushing-2]
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state change to [normal]
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 selected broker 3
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/3
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 state change to [flushing-1]
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 state change to [normal]

• filebeat.yml

#------------------------------- Kafka output ----------------------------------
output.kafka:
  enabled: true

  hosts: ["kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092","kafka-003-cluster.priv.com:9092"]

  topic: "cta_test"

  codec.format:
    string: '"offset":%{[offset]}, "message":%{[message]}'

Answer 1

不是新来的经纪人。

卡夫卡将把advertised.listeners设置返回给客户端。你只提供了一个初始的引导地址给File节拍