Zookeeper:主机名解析失败

Question

我在OpenShift/Kubernetes环境中运行Zookeeper。我将zookeeper设置为StatefulSet，以便可靠地持久化配置数据。

我在zoo.cfg中按主机名配置了三台服务器，但在启动时，主机名解析失败。我验证了在我的集群中使用nslookup确实可以解析主机名。

zoo.cfg：

clientPort=2181
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/log
tickTime=2000
initLimit=10
syncLimit=2000
maxClientCnxns=60
minSessionTimeout= 4000
maxSessionTimeout= 40000
autopurge.snapRetainCount=3
autopurge.purgeInteval=0
server.1=zookeeper-0.zookeeper-headless:2888:3888
server.2=zookeeper-1.zookeeper-headless:2888:3888
server.3=zookeeper-2.zookeeper-headless:2888:3888

我的OpenShift / Kubernetes配置的相关部分：

  # StatefulSet
  - apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      labels:
        app: zookeeper
      name: zookeeper
    spec:
      serviceName: zookeeper-headless
      replicas: 3
      template:
        metadata:
          labels:
            app: zookeeper
        spec:
          containers:
            - image: 172.30.158.156:5000/os-cloud-platform/zookeeper:latest
              name: zookeeper
              ports:
                - containerPort: 2181
                  protocol: TCP
                  name: client
                - containerPort: 2888
                  protocol: TCP
                  name: server
                - containerPort: 3888
                  protocol: TCP
                  name: leader-election
          dnsPolicy: ClusterFirst
          schedulerName: default-scheduler

  # Service
  - apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: zookeeper
      name: zookeeper
    spec:
      ports:
        - name: client
          port: 2181
          protocol: TCP
          targetPort: 2181
      selector:
        app: zookeeper
      sessionAffinity: None
      type: ClusterIP

  - apiVersion: v1
    kind: Service
    metadata:
      name: zookeeper-headless
      labels:
        app: zookeeper
    spec:
      ports:
        - port: 2888
          name: server
        - port: 3888
          name: leader-election
      clusterIP: None
      selector:
        app: zookeeper

不过，OpenShift日志显示的是UnknownHostException：

2017-10-06 10:59:18,289 [myid:] - WARN  [main:QuorumPeer$QuorumServer@155] - Failed to resolve address: zookeeper-2.zookeeper-headless
java.net.UnknownHostException: zookeeper-2.zookeeper-headless: No address associated with hostname
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at java.net.InetAddress.getByName(InetAddress.java:1076)
    at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:148)
    at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.<init>(QuorumPeer.java:133)
    at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:228)
    at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:140)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
...

可能的原因是什么？我验证了主机名(例如zookeeper-2.zookeeper-headless)可以通过nslookup从其他pods获得。

Answer 1

我为这个问题找到了一个可行的解决方案。ZooKeeper在启动时读取集成中的服务器列表，并查找它自己的条目。然后，它使用此条目来确定要侦听的端口和接口。

server.1=zookeeper-0.zookeeper-headless:2888:3888
server.2=zookeeper-1.zookeeper-headless:2888:3888
server.3=zookeeper-2.zookeeper-headless:2888:3888

由于提供的主机名将在此机器上解析为127.0.0.1，因此ZooKeeper将侦听本地环回接口，因此不接受来自其他ZooKeeper服务器的连接。

server.1=0.0.0.0:2888:3888
server.2=zookeeper-1.zookeeper-headless:2888:3888
server.3=zookeeper-2.zookeeper-headless:2888:3888

为了在集群中实现自动化，我编写了一个bash脚本，该脚本将在容器启动时替换一个"own“条目。

编辑：正如评论中询问的，下面是我的ENTRYPOINT脚本，它负责放置myid文件并为每个zoo.cfg设置适当的主机名

#!/bin/bash
# This script extracts the number out of the pod's hostname and sets it as zookeepers id.

# Exact paths may vary according to your setup
MYID_FILE="/var/lib/zookeeper/data/myid"
ZOOCFG_FILE="/conf/zoo.cfg"

# Create myid-file
# Extract only numbers from hostname
id=$(hostname | tr -d -c 0-9)
echo $id > "${MYID_FILE}"

# change own hostname to 0.0.0.0
# otherwise, the own hostname will resolve to 127.0.0.1
# https://stackoverflow.com/a/40750900/5764665
fullHostname="$(hostname).zookeeper-headless"
sed -i -e "s/${fullHostname}/0.0.0.0/g" "${ZOOCFG_FILE}"

echo "Executing $@"
exec "$@"