用棉布在裸金属上安装k8s

Question

在进入生产系统之前，我正在尝试建立一个用于学习和测试的k8s集群。

我已经在Debian11中的裸金属上设置了我的k8s集群

安装后，我可以运行：

$ kubectl get nodes -A
NAME   STATUS   ROLES           AGE   VERSION
km1    Ready    control-plane   22m   v1.26.2
kw1    Ready    worker          21m   v1.26.2

在我看来不错。然而，当我跑步时：

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS                  RESTARTS        AGE
kube-system   calico-kube-controllers-57b57c56f-rp47v   1/1     Running                 0               12m
kube-system   calico-node-m4bsl                         0/1     Init:CrashLoopBackOff   6 (2m54s ago)   8m39s
kube-system   calico-node-tzcp7                         1/1     Running                 0               12m
kube-system   coredns-787d4945fb-cldh2                  1/1     Running                 0               12m
kube-system   coredns-787d4945fb-pcpx8                  1/1     Running                 0               12m
kube-system   etcd-km1                                  1/1     Running                 44              13m
kube-system   kube-apiserver-km1                        1/1     Running                 46              13m
kube-system   kube-controller-manager-km1               1/1     Running                 41              13m
kube-system   kube-proxy-c7m6b                          1/1     Running                 0               12m
kube-system   kube-proxy-sx4hj                          1/1     Running                 0               12m
kube-system   kube-scheduler-km1                        1/1     Running                 41              13m

我看到calico-node-m4bsl不起作用。

1. 这有问题吗？
2. 我是不是做错了什么让这一切发生的事？

如果它能帮助你回答我的问题，这里有一些背景资料：

我的calico.yaml是通过：

$ curl -fLO https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

我对该文件所做的唯一更改是取消注释并设置CALICO_IPV4POOL_CIDR变量：

4601     - name: CALICO_IPV4POOL_CIDR
4602       value: "10.2.0.0/16"

我像这样初始化我的集群：

$ sudo kubeadm init --control-plane-endpoint=km1.lan:6443 --pod-network-cidr=10.2.0.0/16

$ kubectl describe pods -n kube-system calico-node-m4bsl
Name:                 calico-node-m4bsl
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      calico-node
Node:                 kw1/192.168.56.60
Start Time:           Thu, 09 Mar 2023 13:45:09 -0600
Labels:               controller-revision-hash=9889897b6
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          
Status:               Pending
IP:                   192.168.56.60
IPs:
  IP:           192.168.56.60
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  containerd://49d885579623eb69e01288cbfbac8ee06e6a168819764fced9d4a83eba4443c7
    Image:         docker.io/calico/cni:v3.25.0
    Image ID:      docker.io/calico/cni@sha256:a38d53cb8688944eafede2f0eadc478b1b403cefeff7953da57fe9cd2d65e977
    Port:          
    Host Port:     
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 09 Mar 2023 13:45:10 -0600
      Finished:     Thu, 09 Mar 2023 13:45:10 -0600
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:    Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-knm4l (ro)
  install-cni:
    Container ID:  containerd://91271557309b31affd3adc56c8d7ee57c560036d67f787b9e09645926a720b44
    Image:         docker.io/calico/cni:v3.25.0
    Image ID:      docker.io/calico/cni@sha256:a38d53cb8688944eafede2f0eadc478b1b403cefeff7953da57fe9cd2d65e977
    Port:          
    Host Port:     
    Command:
      /opt/cni/bin/install
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 09 Mar 2023 14:06:17 -0600
      Finished:     Thu, 09 Mar 2023 14:06:18 -0600
    Ready:          False
    Restart Count:  9
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:      Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:                 Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-knm4l (ro)
  mount-bpffs:
    Container ID:
    Image:         docker.io/calico/node:v3.25.0
    Image ID:
    Port:          
    Host Port:     
    Command:
      calico-node
      -init
      -best-effort
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    
    Mounts:
      /nodeproc from nodeproc (ro)
      /sys/fs from sys-fs (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-knm4l (ro)
Containers:
  calico-node:
    Container ID:
    Image:          docker.io/calico/node:v3.25.0
    Image ID:
    Port:           
    Host Port:      
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:            Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      CALICO_IPV6POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                      Optional: false
      FELIX_VXLANMTU:                       Optional: false
      FELIX_WIREGUARDMTU:                   Optional: false
      CALICO_IPV4POOL_CIDR:               10.100.0.0/16
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_HEALTHENABLED:                true
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/bpf from bpffs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-knm4l (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sys-fs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  bpffs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/bpf
    HostPathType:  Directory
  nodeproc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  kube-api-access-knm4l:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  23m                   default-scheduler  Successfully assigned kube-system/calico-node-m4bsl to kw1
  Normal   Pulled     23m                   kubelet            Container image "docker.io/calico/cni:v3.25.0" already present on machine
  Normal   Created    23m                   kubelet            Created container upgrade-ipam
  Normal   Started    23m                   kubelet            Started container upgrade-ipam
  Normal   Pulled     22m (x5 over 23m)     kubelet            Container image "docker.io/calico/cni:v3.25.0" already present on machine
  Normal   Created    22m (x5 over 23m)     kubelet            Created container install-cni
  Normal   Started    22m (x5 over 23m)     kubelet            Started container install-cni
  Warning  BackOff    3m50s (x94 over 23m)  kubelet            Back-off restarting failed container install-cni in pod calico-node-m4bsl_kube-system(80c1a06b-7522-4df6-8c5e-e7e1beb41cd0)

Answer 1

怀疑根本原因是kubelet启动了portmap的多个实例，这些实例阻止install-cni容器复制该可执行文件并完成安装到与剪纸节点共享的卷中。这似乎是因为kubelet和calico都在争夺对同一个可执行文件(即/home/kubernetes/bin/portmap )的访问权限。有关细节，请参阅支持主机端口。

正如您所描述的(calico容器init故障)，棉布节点荚无法恢复。因此，依赖于网络策略的用户工作负载也无法启动。

修改kube-system的棉布节点，在后台YAML中设置UPDATE_CNI_BINARIES="false"，包括如下所示；

- env
 - name: UPDATE_CNI_BINARIES
   value: "false"

此外，检查您可能面临一个临时的资源过载问题，因为一个活动高峰。尝试更改periodSeconds或timeoutSeconds，以便给应用程序足够的响应时间。