卡夫卡问题警告[生产者客户端=控制台-生产者]连接到node.......java.net.UnknownHostException的错误

Question

我想把卡夫卡安排在库伯奈特。一切都开始的很好，包括动物园管理员和卡夫卡服务器，因为他们的日志，但当我试图产生一个主题的消息，它说“错误连接到节点”。谁能看看下面的yaml文件和错误，并告诉我是什么问题。谢谢。以下是详细信息：

zookeeper/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper-cluster1
  namespace: default
  labels:
    app: zookeeper-cluster1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-cluster1
  template:
    metadata:
      labels:
        name: zookeeper-cluster1
        app: zookeeper-cluster1
    spec:
      hostname: zookeeper-cluster1
      containers:
        - name: zookeeper-cluster1
          image: wurstmeister/zookeeper:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888

zookeeper/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: zookeeper-cluster1
  namespace: default
  labels:
    app: zookeeper-cluster1
spec:
  type: NodePort
  selector:
    app: zookeeper-cluster1
  ports:
    - name: zookeeper-cluster1
      protocol: TCP
      port: 2181
      targetPort: 2181
    - name: zookeeper-follower-cluster1
      protocol: TCP
      port: 2888
      targetPort: 2888
    - name: zookeeper-leader-cluster1
      protocol: TCP
      port: 3888
      targetPort: 3888

kafka/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-cluster
  namespace: default
  labels:
    app: kafka-cluster
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-cluster
  template:
    metadata:
      labels:
        name: kafka-cluster
        app: kafka-cluster
    spec:
      hostname: kafka-cluster
      containers:
        - name: kafka-cluster
          image: wurstmeister/kafka:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://kafka-cluster:9092
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: zookeeper-cluster1:2181
            - name: KAFKA_LISTENERS
              value: PLAINTEXT://0.0.0.0:9092
          ports:
            - containerPort: 9092

kafka/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: kafka-cluster
  namespace: default
  labels:
    app: kafka-cluster
spec:
  type: NodePort
  selector:
    app: kafka-cluster
  ports:
    - name: kafka-cluster
      protocol: TCP
      port: 9092
      targetPort: 9092

kafkacat输出

kafkacat -b 172.17.0.2:30511 -L

Metadata for all topics (from broker -1: 172.17.0.2:30511/bootstrap):
 1 brokers:
  broker 1001 at kafka-cluster:9092
 0 topics:

产生消息时错误

    bin/kafka-console-producer.sh --topic admintome-test1 --bootstrap-server 172.17.0.2:30511
    >as
[2021-02-10 15:58:04,696] WARN [Producer clientId=console-producer] Error connecting to node kafka-cluster:9092 (id: 1001 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka-cluster: Name or service not known
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
    at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:74)
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1131)
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1019)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:542)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)
    at java.base/java.lang.Thread.run(Thread.java:834)

kafka-服务器日志

[2021-02-10 10:50:01,044] INFO [TransactionCoordinator id=1001] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2021-02-10 10:50:01,091] INFO [Transaction Marker Channel Manager 1001]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2021-02-10 10:50:01,091] INFO [TransactionCoordinator id=1001] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2021-02-10 10:50:01,312] INFO Updated cache from existing <empty> to latest FinalizedFeaturesAndEpoch(features=Features{}, epoch=0). (kafka.server.FinalizedFeatureCache)
[2021-02-10 10:50:01,508] INFO [ExpirationReaper-1001-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2021-02-10 10:50:01,641] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2021-02-10 10:50:01,701] INFO [SocketServer brokerId=1001] Starting socket server acceptors and processors (kafka.network.SocketServer)
[2021-02-10 10:50:01,708] INFO [SocketServer brokerId=1001] Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
[2021-02-10 10:50:01,708] INFO [SocketServer brokerId=1001] Started socket server acceptors and processors (kafka.network.SocketServer)
[2021-02-10 10:50:01,716] INFO Kafka version: 2.7.0 (org.apache.kafka.common.utils.AppInfoParser)
[2021-02-10 10:50:01,716] INFO Kafka commitId: 448719dc99a19793 (org.apache.kafka.common.utils.AppInfoParser)
[2021-02-10 10:50:01,716] INFO Kafka startTimeMs: 1612954201709 (org.apache.kafka.common.utils.AppInfoParser)
[2021-02-10 10:50:01,718] INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
[2021-02-10 10:50:02,005] INFO [broker-1001-to-controller-send-thread]: Recorded new controller, from now on will use broker 1001 (kafka.server.BrokerToControllerRequestThread)

动物园管理员日志

2021-02-10 10:49:44,841 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2021-02-10 10:49:44,845 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
2021-02-10 10:49:59,583 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /172.18.0.4:60322
2021-02-10 10:49:59,588 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /172.18.0.4:60322
2021-02-10 10:49:59,590 [myid:] - INFO  [SyncThread:0:FileTxnLog@213] - Creating new log file: log.1
2021-02-10 10:49:59,601 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@694] - Established session 0x10007ad45a10000 with negotiated timeout 18000 for client /172.18.0.4:60322
2021-02-10 10:49:59,669 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x10007ad45a10000 type:create cxid:0x2 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers
2021-02-10 10:49:59,679 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x10007ad45a10000 type:create cxid:0x6 zxid:0x7 txntype:-1 reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config
2021-02-10 10:49:59,686 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x10007ad45a10000 type:create cxid:0x9 zxid:0xa txntype:-1 reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin
2021-02-10 10:49:59,948 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x10007ad45a10000 type:create cxid:0x17 zxid:0x15 txntype:-1 reqpath:n/a Error Path:/cluster Error:KeeperErrorCode = NoNode for /cluster
2021-02-10 10:50:01,804 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@596] - Got user-level KeeperException when processing sessionid:0x10007ad45a10000 type:multi cxid:0x3e zxid:0x1e txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election

Answer 1

对卡夫卡吐露的一个简短的解释：

1. listeners --由卡夫卡中的嵌入式码头web服务器来绑定。这个jetty服务器用于提供REST，该API为Kafka工作人员提供控制平面。如果希望kafka绑定到本地主机，则可以将此设置中的主机名保留为空(通过调用InetAddress.getLocalHost().getCanonicalHostName()、java、api)
2. advertised.listeners：，此地址由每个kafka代理发布给动物园管理员。如果未设置此设置，则此处将使用侦听器的值，并将其发布给动物园管理员。这是此设置通知他人的唯一目的。Kafka客户端使用发布给动物园管理员的“advertised.listeners”设置(as / broker /ids//#端点)与Kafka broker对话。见(https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic)

NOTE: The exception is ZooKeeper-based consumers. These consumers retrieve the broker registration information directly from ZooKeeper and will choose the first listener with PLAINTEXT as the security protocol (the only security protocol they support).

在您的例子中，您需要同时设置listeners和advertised.listeners。对于外部访问(即k8外部)，请将advertised.listerners地址设置为节点ip或负载均衡器ip，并将端口设置为nodePort。当提供多个advtised.listeners地址时，请确保逗号分隔的url中的第一个地址具有用于外部访问的ip，以便在动物园管理员中很好地填充以供客户使用。

示例znode输出(客户端将使用明文://172.28.120.125:30029)，它在k8环境之外是可解析的。(Config: advertised.listeners=OUTSIDE://172.28.120.125:30029，INSIDE://:9092)

zk: localhost:2181(已连接) 11 get /brokers/ids/0 {“功能”：{}，"listener_security_protocol_map":{"OUTSIDE":"PLAINTEXT"，“ids”：“明文”}，“终结点”：“外部://172.28.120.125:30029”，“内部://kafka-6877666f69-br7c4:9092”，"jmx_port":-1，“端口”：30029，“主机”：“172.28.120.125”，“版本”：5，“时间戳”：“1638483151305”}当advertised.listeners设置为localhost或空(例如明文://:9092)时，豆荚的主机名将被公布，因此您将看到该错误(UnknownHostException)。这就是在动物园管理员中宣传主机名的方式。在本例中，明文://Kafka-55d5d99c89-ksr5n:9092将被发布广告，这在k8环境之外是无法解决的。(Config: adsed.listents=INSIDE://:9092，OUTSIDE://172.28.120.125:30029)

zk: localhost:2181(已连接)3 get /brokers/ids/0 {“功能”：{}，"listener_security_protocol_map":{"INSIDE":"PLAINTEXT"，“get”：“明文”}，"endpoints":"INSIDE://kafka-55d5d99c89-ksr5n:9092"，外部://172.28.120.125:9094，"jmx_port":-1，“端口”：9092，“主机”：“kafka-55d5d99c89-ksr5n”，“版本”：5，“时间戳”：“1638517892783”}

检查下面的示例配置。

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  namespace: my-namespace
  labels:
    name: kafka
spec:
  selector:
    app: kafka
    id: "0"  
  ports:
  - port: 30029
    targetPort: 30029
    nodePort: 30029
    name: kafka-port
    protocol: TCP 
  type: NodePort     
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: kafka
  namespace: my-namespace
  labels:
    name: kafka
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka 
        env:
        - name: KAFKA_ADVERTISED_LISTENERS
          value: OUTSIDE://172.28.120.125:30029,INSIDE://:9092
        - name: KAFKA_LISTENERS
          value: INSIDE://:9092,OUTSIDE://:30029
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: INSIDE
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper:2181
        - name: KAFKA_BROKER_ID
          value: "0"
      volumes:
      - name: kafka-pv
        persistentVolumeClaim: 
          claimName: my-pvc