卡夫卡的文件:未能连接到经纪人

Question

我是apache环境中的新成员，目前我正在尝试将日志数据从filebeat生产者发送到kafka broker。

环境：

1. kafka 2.11 (通过ambari安装)
2. 文件质量7.4.2 (窗口)

我试着把文件记录发送到ambari，我启动了kafka服务器并创建了名为"test“的主题，并将其列在列表中。我对卡夫卡经纪人的港口很困惑。在一些教程中，我看到他们使用的是9092而不是2181。那么，现在我应该使用哪个端口来发送文件记录呢？

这是我的filebeat.conf

filebeat.inputs:
    - type: log
      paths:
       - C:/Users/A/Desktop/DATA/mailbox3.csv
output.kafka:
  hosts: ["x.x.x.x:9092"]
  topic: "test"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

结果

2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)

2020-06-10T09:00:32.466+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:34.478+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining)

2020-06-10T09:00:34.729+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)

2020-06-10T09:00:36.989+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

这让我想知道我是否真的有9092港口。因此，我查看了server.properties.Some，我最关心的是：

port=6667
listeners=PLAINTEXT://x.x.x.x:6667

然后，我再次尝试执行filebeat.conf并将端口9092更改为6667，结果如下

2020-06-10T09:18:01.448+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667

2020-06-10T09:18:01.450+0700    INFO    kafka/log.go:53 producer/broker/1001 starting up

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [open] on test/0

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/leader/test/0 selected broker 1001

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.452+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:264     finished kafka batch
2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:278     Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.454+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [normal]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 abandoning broker 1001

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/broker/1001 shut down

问题

发生了什么？应该使用哪个端口？每个端口的用途是什么？

任何回应都将是非常感谢的。谢谢

更新

根据这个来源，正确的源代码是6667，因为kafka是通过ambari安装的。

Answer 1

对可以使用的端口没有限制，它只取决于可用性。在第一种情况下，正如您所说的，代理可以在6667启动，因此在9092上没有运行任何进程。

2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53未能连接到代理x.x:6667:拨号tcp:查找x.x:没有这样的主机

接下来，当您提到advertised.listeners属性时，您应该确保您在advertised.listeners中提到的IP是分配给该机器的IP。您不能分配1.1.1.1:9092 (仅举几个例子)。

执行ifconfig (linux)、ipconfig (windows)并在从应用程序机器访问的网络接口上查看计算机的IP。在linux中，它主要是eth0。

此IP必须可从运行应用程序的计算机访问。因此，您的应用程序正在运行的机器应该能够解析该IP。您还可能希望检查您的Kafka代理程序与运行应用程序的机器之间的网络连接。