Kubernetes HA:法兰绒抛出SubnetManager错误

Question

我一直在遵循https://kubernetes.io/docs/setup/independent/high-availability/提供的步骤来创建HA集群。我使用的是CoreOS节点(VERSION=1688.5.3)和Kubernetes版本v1.10。

我遵循了在主节点上运行所有三个etcd的选项。对于负载均衡器，我使用了在https://github.com/alterway/docker-keepalived中找到的容器化保存。上传到容器化保存的keepalived.conf文件如k8s HA指南本身所示。

当我达到配置CNI网络(https://kubernetes.io/docs/setup/independent/high-availability/#install-cni-network)的步骤时，法兰绒-ds吊舱进入CrashLoopBackoff时出错：“未能创建SubnetManager:错误检索‘kube-system/SubnetManager ds-fjn6w’的荚规范: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-fjn6w：拨号tcp 10.96.0.1:443: i/o超时值”

这到底是怎么回事？下面是运行法兰绒-ds吊舱的主节点的iptables：

The flannel pod is trying to retrieve its configuration from the API server using the service-IP 10.96.0.1, which is supposed to get DNAT to node IPs
    -A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
    -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-SIIK55AX7MK5ONR7
    -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-GBLS75FLCCJBNQB6
    -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-2CDZMOLH2PKAG52U

But I don’t see these rules being triggered at all.
    0     0 KUBE-SEP-SIIK55AX7MK5ONR7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ statistic mode random probability 0.33332999982
    0     0 KUBE-SEP-GBLS75FLCCJBNQB6  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-2CDZMOLH2PKAG52U  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */


Curl to the service IP does not work, however a curl request to the kubernetes cluster IP gets a response:

    master # curl -k https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/
    curl: (7) Failed to connect to 10.96.0.1 port 443: Connection timed out

    master # curl -k https://10.106.73.226:6443/api/v1/namespaces/kube-system/pods/
    {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {

      },
      "status": "Failure",
      "message": "pods is forbidden: User \"system:anonymous\" cannot list pods in the namespace \"kube-system\"",
      "reason": "Forbidden",
      "details": {
        "kind": "pods"
      },
      "code": 403

Also note, the service endpoints have been set correctly to the cluster IP:

    master # kubectl describe svc kubernetes
    Name:              kubernetes
    Namespace:         default
    Labels:            component=apiserver
                       provider=kubernetes
    Annotations:       <none>
    Selector:          <none>
    Type:              ClusterIP
    IP:                10.96.0.1
    Port:              https  443/TCP
    TargetPort:        6443/TCP
    Endpoints:         10.106.73.226:6443
    Session Affinity:  ClientIP
    Events:            <none>

    master # kubectl cluster-info
    Kubernetes master is running at https://10.106.73.226:6443
    KubeDNS is running at https://10.106.73.226:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

我尝试添加DNAT来手动将集群IP映射到服务IP，但help...although似乎不确定是否将规则添加到正确的iptable链中。

编辑1-完整的iptables

master ~ # iptables -S -t nat 
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODEPORTS
-N KUBE-POSTROUTING
-N KUBE-SEP-PE4UL45OLJLNLYYS
-N KUBE-SERVICES
-N KUBE-SVC-NPX46M4PTMTKRN6Y
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -d 10.96.0.1/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.106.73.226:6443
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-PE4UL45OLJLNLYYS -s 10.106.73.226/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-PE4UL45OLJLNLYYS -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-PE4UL45OLJLNLYYS --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.106.73.226:6443
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-PE4UL45OLJLNLYYS --mask 255.255.255.255 --rsource -j KUBE-SEP-PE4UL45OLJLNLYYS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-PE4UL45OLJLNLYYS

注意:我手动添加了规则-A PREROUTING -d 10.96.0.1/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.106.73.226:6443，希望将10.96.0.1映射到apiserver IP，但这并没有改变卷曲请求或法兰绒荚的行为。

主舱的当前状态：

master ~ # kubectl get pods -o wide --all-namespaces
NAME                                  READY     STATUS              RESTARTS   AGE       IP              NODE
etcd-master                           1/1       Running             0          13d       10.106.73.226   master
kube-apiserver-master                 1/1       Running             0          13d       10.106.73.226   master
kube-controller-manager-master        1/1       Running             1          13d       10.106.73.226   master
kube-dns-86f4d74b45-dkzlk             0/3       ContainerCreating   0          13d       <none>          master
kube-flannel-ds-j5fxd                 0/1       CrashLoopBackOff    3550       13d       10.106.73.226   master
kube-proxy-pml47                      1/1       Running             0          13d       10.106.73.226   master
kube-scheduler-master                 1/1       Running             0          13d       10.106.73.226   master

Answer 1

您的所有设置看起来都不错，包括路由和systctl值。

我唯一能猜到的就是防火墙规则中的某个问题。请确保您接受Forward链中的流量转发。

你可以这样检查：

1. 阻止K8s。
2. 停止防火墙。
3. 停下码头。
4. 写入文件/var/lib/iptables/rules-save内容(如果存在该文件，则重写该内容)： *过滤器:输入接受0:0 :向前接受0:0 :输出接受0:0
5. 启动防火墙。
6. 启动码头。
7. 启动K8s。
8. 检查服务。

这是我能想象出为什么你对服务有问题的唯一原因。