如何使用nodeport服务类型在k8s环境中创建redis集群？

Question

我尝试使用“k8s”类型的服务在NodePort环境中创建一个redis集群。更具体地说，我想在两个不同的k8s集群上组成一个redis集群。

当我使用LoadBalancer(外部IP)作为服务类型时，集群已经成功地实现了。问题是NodePort。

在我命令redis-cli -群集创建之后，它会插入“等待集群加入”。

下面是集群创建命令的日志。我部署了4个领袖舱和4个奴隶舱，并配备了单独的节点端口服务。

root@redis-leader00-5fc546c4bd-28x8w:/data# redis-cli -a mypassword --cluster create --cluster-replicas 1 \
> 192.168.9.194:30030 192.168.9.199:30031 192.168.9.194:30032 192.168.9.199:30033 \
> 192.168.9.199:30030 192.168.9.194:30031 192.168.9.199:30032 192.168.9.194:30033
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 8 nodes...
Master[0] -> Slots 0 - 4095
Master[1] -> Slots 4096 - 8191
Master[2] -> Slots 8192 - 12287
Master[3] -> Slots 12288 - 16383
Adding replica 192.168.9.199:30030 to 192.168.9.194:30030
Adding replica 192.168.9.194:30033 to 192.168.9.199:30031
Adding replica 192.168.9.199:30032 to 192.168.9.194:30032
Adding replica 192.168.9.194:30031 to 192.168.9.199:30033
M: 94bf3c6760e6b3a91c408eda497822b4961e8d82 192.168.9.194:30030
   slots:[0-4095] (4096 slots) master
M: 31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031
   slots:[4096-8191] (4096 slots) master
M: 0738d1e1a677352fc3b0b3600a67d837b795fa8a 192.168.9.194:30032
   slots:[8192-12287] (4096 slots) master
M: 7dd7edbfab6952273460778d1f140b0716118042 192.168.9.199:30033
   slots:[12288-16383] (4096 slots) master
S: 17e044681319d0a05bd92deeb4ead31c0cd468e2 192.168.9.199:30030
   replicates 94bf3c6760e6b3a91c408eda497822b4961e8d82
S: 9c9e47ec566ac781e8e3dcb51398a27d1da71004 192.168.9.194:30031
   replicates 7dd7edbfab6952273460778d1f140b0716118042
S: b8f7028b56f96565a91fdb442c94fbedcee088c2 192.168.9.199:30032
   replicates 0738d1e1a677352fc3b0b3600a67d837b795fa8a
S: e4c9ffdf67e2b2ef9f840618110738358bde52d5 192.168.9.194:30033
   replicates 31f4a9604b15109316f91956aa4a32b0c6952a4d
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...........................................

奇怪的是，其他红色容器收到了信号，但没有进展。下面是其他红色容器的原木。

.... // Some logs for initializing redis
1:M 20 Jan 2022 06:09:12.055 * Ready to accept connections
1:M 20 Jan 2022 06:13:41.263 # configEpoch set to 5 via CLUSTER SET-CONFIG-EPOCH

我认为沟通是成功的，但流言港没有正常运作。因此，我修改了redis.conf并设置了集群公告总线端口，但它也没有工作。

如何使用nodeport类型的服务组成redis集群？

请参考.yaml文件中的一个

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-leader00
  namespace: redis
  labels:
    app: redis-cluster
    leader: "00"
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${HOST_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${HOST_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${HOST_IP}/" ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |+
    bind 0.0.0.0
    cluster-enabled yes
    cluster-require-full-coverage no
    cluster-node-timeout 15000
    cluster-config-file /data/nodes.conf
    cluster-migration-barrier 1
    appendonly no
    save ""
    protected-mode no
    requirepass "mypassword"
    masterauth "mypassword"
    cluster-announce-ip 192.168.9.194
    cluster-announce-port 30030
    cluster-announce-bus-port 31030
---
apiVersion: v1
kind: Service
metadata:
  name: redis-leader00
  namespace: redis
  labels:
    app: redis
    role: leader
    tier: backend
    leader: "00"
spec:
  ports:
  - port: 6379
    targetPort: 6379
    nodePort: 30030
    name: client
  - port: 16379
    targetPort: 16379
    nodePort: 31030
    name: gossip
  selector:
    app: redis
    role: leader
    tier: backend
    leader: "00"
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-leader00
  namespace: redis
  labels:
    app: redis
    role: leader
    tier: backend
    leader: "00"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
      leader: "00"
  template:
    metadata:
      labels:
        app: redis
        role: leader
        tier: backend
        leader: "00"
    spec:
      containers:
      - name: leader
        image: docker.io/redis:6.0.5
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        args: ["--requirepass", "mypassword"]
        command: ["/conf/fix-ip.sh", "redis-server", "/conf/redis.conf"]
        env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
      volumes:
      - name: conf
        configMap:
          name: redis-cluster-leader00
          defaultMode: 0755

在我命令redis-cli创建之后，还可以看到容器中的nodes.conf文件。

root@redis-leader01-87ccb466-bsnq4:/data# cat nodes.conf
31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031@31031 myself,master - 0 0 2 connected 4096-8191
vars currentEpoch 2 lastVoteEpoch 0

Answer 1

还有一个人已经发现了这个问题。这就是他提出的问题。https://github.com/redis/redis/issues/6432

造成问题的原因是集群会议不使用集群公告总线端口.实际上，在clusterManagerNode结构中甚至没有存储集群宣布总线端口的变量。

对于这个问题，已经有了公共关系。https://github.com/redis/redis/pull/6442

它还没有被合并或关闭，所以这是一个持续的问题.

Answer 2

我不确定您创建Redis集群所遵循的实际过程，不过，我建议您查看舵图，以便在K8s上部署Redis集群。

使用helm图表，可以轻松地在K8s上管理和部署Redis集群。

https://github.com/bitnami/charts/tree/master/bitnami/redis

要部署图表，只需运行命令：

helm install my-release bitnami/redis

在节点端口一侧，一旦部署了舵图，您就可以编辑“否则”的服务类型，还可以更新第一个舵图，然后将这些更改应用到K8s中。

这将为Redis服务在K8s上创建节点端口。

您可以在这里找到服务模板：https://github.com/bitnami/charts/tree/master/bitnami/redis/templates