HA kubernetes集群:意外地在1个主节点上重置kubeadm，重新加入群集时拒绝连接

Question

我已经设置了一个包含2个主节点(cp01 192.168.1.42，cp02 192.168.1.46)和4个工作节点的kubernetes集群，它们使用have实现，并作为集群中的静态荚保持运行，内部etcd集群。出于一些愚蠢的原因，我意外地在cp01上重置了cp01。现在，我尝试使用kubeadm命令重新加入集群，但是我一直得到拨号TCP192.168.1.49:8443: connect: connection拒绝连接，其中192.168.1.49是LoadBalancer IP。请帮帮我！下面是当前的配置。

/etc/haproxy/haproxy.cfg on on 02

defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s
frontend apiserver
    bind *.8443
    mode tcp
    option tcplog
    default_backend apiserver
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance roundrobin
        default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
        #server master01 192.168.1.42:6443 check     ***the one i accidentally resetted
        server master02 192.168.1.46:6443 check

cp02 02上的/etc/持存/保持.cp02

global_defs {
    router_id LVS_DEVEL
    script_user root
    enable_script_security
    dynamic_interfaces
}
vrrp_script check_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 3
    weight -2
    fall 10
    rise 2
}
vrrp_instance VI_l {
    state BACKUP
    interface ens192
    virtual_router_id 51
    priority 101
    authentication {
        auth_type PASS
        auth_pass ***
    }
    virtual_ipaddress {
        192.168.1.49/24
    }
    track_script {
        check_apiserver
    }
}

<#>cluster kubeadm-config

apiVersion: v1
data:
    ClusterConfiguration: |
        apiServer:
            extraArgs:
                authorization-mode: Node,RBAC
            timeoutForControlPlane: 4m0s
        apiVersion: kubeadm.k8s.io/v1beta2
        certificatesDir: /etc/kubernetes/pki
        clusterName: kubernetes
        controlPlaneEndpoint: 192.168.1.49:8443
        controllerManager: {}
        dns:
            type: CoreDNS
        etcd:
            local:
                dataDir: /var/lib/etcd
        imageRepository: k8s.gcr.io
        kind: ClusterConfiguration
        kubernetesVersion: v1.19.2
        networking:
            dnsDomain: cluster.local
            podSubnet: 10.244.0.0/16
            serviceSubnet: 10.96.0.0/12
        scheduler: {}
    ClusterStatus: |
        apiEndpoints:
            cp02:
                advertiseAddress: 192.168.1.46
                bindPort: 6443
        apiVersion: kubeadm.k8s.io/v1beta2
        kind: ClusterStatus
...

kubectl集群-info

Kubernetes master is running at https://192.168.1.49:8443
KubeDNS is running at https://192.168.1.49:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

More信息

1. 集群由cp01上的--上传证书初始化。
2. 我从集群中抽干并删除了cp01。
3. kubeadm join --token ... --discovery-token-ca-cert-hash ... --control-plane --certificate-key ...命令返回:错误执行阶段预飞:无法获取kubeadm-config ConfigMap:未能获取配置映射: get "https://192.168.1.49:8443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s“：拨号tcp 192.168.1.49:8443:连接:连接被拒绝
4. kubectl exec -n kube-system -it etcd-cp02 -- etcdctl --endpoints=https://192.168.1.46:2379 --key=/etc/kubernetes/pki/etcd/peer.key --cert=/etc/kubernetes/pki/etcd/peer.crt --cacert=/etc/kubernetes/pki/etcd/ca.crt member list返回：.，started，cp02，https://192.168.1.46:2380，https://192.168.1.46:2379，false
5. kubectl describe pod/etcd-cp02 -n kube-system：...集装箱ID:码头://.图片: k8s.gcr.io/etcd:3.4.13-0图像ID: docker://.端口：--cert-file=/etc/kubernetes/pki/etcd/server.crt：主机端口：命令: etcd --广告-client-urls=https://192.168.1.46:2379 --key-file=/etc/kubernetes/pki/etcd/server.key --client-cert-auth=true -data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://192.168.1.46:2380 -初始化-群集=https://192.168.1.42:2380，cp02=https://192.168.1.46:2380-初始-群集-状态=现有--key-file=/etc/kubernetes/pki/etcd/server.key-侦听-客户端-urls=https://127.0.0.1:2379，https://192.168.1.46:2379-听-度量-urls=http://127.0.0.1:2381-听-对等-urls=https://192.168.1.46:2380-名称=cp02 02C24-对等客户端证书-auth=真正的--peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt -快照-计数=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt .
6. 在运行cp01:/etc/kubernetes/pki之前，尝试将证书复制到kubeadm join 192.168.1.49:8443 --token ... --discovery-token-ca-cert-hash，但返回了相同的错误。#文件复制到cp01 ca.crt ca.key sa.key sa.pub前端-代理-ca.crt前端-代理-ca.key etcd/ca.crt etcd/ca.key

Troubleshoot网络

1. 能够在cp01上ping 192.168.1.49
2. nc -v 192.168.1.49 8443 on cp01返回Ncat: Connection refused.
3. curl -k https://192.168.1.49:8443/api/v1...在cp02和工作节点上工作(返回代码403，这应该是正常的)。
4. /etc/cni/net.d/在cp01上删除
5. 手动清除cp01上带有“KUBE”或“cali”的iptables规则。
6. 在cp01和cp02上都禁用了firewalld。
7. 我尝试加入一个新服务器cp03 192.168.1.48，并遇到了相同的拨号TCP192.168.1.49:8443: connect: connection拒绝错误。
8. netstat -tlnp | grep 8443 on cp02返回:TCP0 0.0.0.0:8443 0.0.0.0:* LISTEN 27316/haproxy
9. nc -v 192.168.1.46 6443 on cp01和cp03返回: Ncat:连接到192.168.1.46:6443

如有任何建议/指导，我将不胜感激，因为我在这里不知所措。我认为这可能是由于cp02上的网络规则所致，但我不知道如何检查。谢谢你！！

Answer 1

我进入ip a的时候发现了什么问题。意识到ens192 on cp01仍然包含二级ip地址192.168.1.49。

简单地说，ip addr del 192.168.1.49/24 dev ens192、kubeadm join...和cp01能够成功地重新加入集群。不敢相信我错过了..。