基于Calico的pod不会出现

Question

我正在使用calico设置一个pod，但它一直失败，出现一些授权错误。默认情况下，以下是我的系统的节点cidr：

[root@k8master-1 ~]# kubeadm config view | grep Subnet
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12

我已经使用以下步骤设置了ippool：

https://docs.projectcalico.org/getting-started/kubernetes/flannel/flannel

IP池创建

- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    name: rack-ip-pool 
  spec:
    blockSize: 26
    cidr: 10.244.1.0/24
    ipipMode: Never
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: Never

Ip池列表

[root@k8master-1 ~]# calicoctl get ippool -o wide
NAME          CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   SELECTOR   
rack-ip-pool   10.244.1.0/24   true   Never      Never       false      all()   

Pod Yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: testcalico
  labels:
    cracklerack: "1"
spec:
  serviceName: testcalico-svc
  selector:
    matchLabels:
      cracklerack: "1"
  template:
    metadata:
      labels:
        cracklerack: "1"
      annotations:
       cni.projectcalico.org/ipv4pools: "[\"rack-ip-pool\"]"
    spec:
      runtimeClassName: kata-containers
      containers:
      - name: testcalico
        image: cracklelinux:7
        ports:
        - containerPort: 80
        command: [/usr/sbin/init]
        securityContext:
          privileged: true
---
apiVersion: v1
kind: Service
metadata:
 name: testcalico-svc
spec:
 clusterIP: None
 selector:
   cracklerack: "1"

当我创建一个pod时，它抛出了以下错误：

错误

 Warning  FailedCreatePodSandBox  112s  kubelet, k8worker-1  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_xxxxx-0_default_45357eab-bf40-4fe7-a470-da42c9668116_0(579e2c258154fcdc2e85df4a1e35264ea9550b0dd1c4384331abc471f552456d): connection is unauthorized: ipamconfigs.crd.projectcalico.org "default" is forbidden: User "system:serviceaccount:kube-system:canal" cannot get resource "ipamconfigs" in API group "crd.projectcalico.org" at the cluster scope

Answer 1

看起来你有一个基于角色的访问控制问题，你的pod不能读取Kubernetes IPAMConfig CRD。

我查看了https://docs.projectcalico.org/manifests/canal.yaml的清单，发现它缺少几个基于角色的访问控制ClusterRoles中的ipamconfigs。因此，您可以继续尝试添加它们。

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Nodes are watched to monitor for deletions.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - watch
      - list
      - get
  # Pods are queried to check for existence.
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
  # IPAM resources are manipulated when nodes are deleted.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
    verbs:
      - list
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
      - ipamconfigs ? add here
 ...

然后是另一个ClusterRole：

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
...
  # Calico monitors various CRDs for config.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - ipamblocks
      - ipamconfigs ? add here
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
      - clusterinformations
      - hostendpoints
      - blockaffinities
    verbs:
      - get
      - list
      - watch
  # Calico must create and update some CRDs on startup.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - ipamconfigs ? just in case
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
...

然后运行：

kubectl apply -f canal.yaml

应用此命令后，您可能需要重新启动集群(至少在我的minikube上需要)。

Answer 2

我使用了下面的conf文件，它起作用了：

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
      # Used to discover Typhas.
      - get
  # Pod CIDR auto-detection on kubeadm needs access to config maps.
  - apiGroups: [""]
    resources:
      - configmaps
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch
      # Calico stores some configuration information in node annotations.
      - update
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
  # Used by Calico for policy information.
  - apiGroups: [""]
    resources:
      - pods
      - namespaces
      - serviceaccounts
    verbs:
      - list
      - watch
  # The CNI plugin patches pods/status.
  - apiGroups: [""]
    resources:
      - pods/status
    verbs:
      - patch
  # Calico monitors various CRDs for config.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - ipamblocks
      - ipamconfigs
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
      - clusterinformations
      - hostendpoints
      - blockaffinities
    verbs:
      - get
      - list
      - watch
  # Calico must create and update some CRDs on startup.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - ipamblocks
      - ipamconfigs
      - blockaffinities
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
  # Calico stores some configuration information on the node.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  # These permissions are only required for upgrade from v2.6, and can
  # be removed after upgrade or on fresh installations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - bgpconfigurations
      - bgppeers
    verbs:
      - create
      - update

同一文件中的另一个块：

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Nodes are watched to monitor for deletions.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - watch
      - list
      - get
  # Pods are queried to check for existence.
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
  # IPAM resources are manipulated when nodes are deleted.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
    verbs:
      - list
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
      - ipamconfigs
    verbs:
      - get
      - list
      - create
      - update
      - delete
  # kube-controllers manages hostendpoints.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - hostendpoints
    verbs:
      - get
      - list
      - create
      - update
      - delete
  # Needs access to update clusterinformations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - clusterinformations
    verbs:
      - get
      - create
      - update
  # KubeControllersConfiguration is where it gets its config
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - kubecontrollersconfigurations
    verbs:
      # read its own config
      - get
      # create a default if none exists
      - create
      # update status
      - update
      # watch for changes
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---