Kubernetes Affinity阻止Jenkins Workers在Main上运行

Question

Jenkins在EKS上运行，在Jenkins main和worker pods上都有亲和性规则。

这个想法是为了防止Jenkins worker pod在运行Jenkins main pod的相同EKS worker节点上运行。

以下规则有效，直到达到资源限制，此时Jenkins worker pod被调度到与Jenkins master pod相同的EKS worker节点上。

是否有亲和性/反亲和性规则来防止这种情况发生？

Jenkins main的规则如下：

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions: # assign to eks apps worker group
            - key: node.app/group
              operator: In
              values:
                - apps
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions: # don't assign to a node running jenkins main
          - key: app.kubernetes.io/name
            operator: In
            values:
              - jenkins
          - key: app.kubernetes.io/component
            operator: In
            values:
              - main
        topologyKey: kubernetes.io/hostname
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions: # try not to assign to a node already running a jenkins worker
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - jenkins
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - worker
          topologyKey: kubernetes.io/hostname

Jenkins worker的规则如下：

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions: # assign to eks apps worker group
          - key:  node.app/group
            operator: In
            values:
              - apps
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions: # don't assign to a node running jenkins main
          - key: app.kubernetes.io/name
            operator: In
            values:
              - jenkins
          - key: app.kubernetes.io/component
            operator: In
            values:
              - main
        topologyKey: kubernetes.io/hostname
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions: # try not to assign to a node already running a jenkins worker
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - jenkins
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - worker
          topologyKey: kubernetes.io/hostname

Answer 1

因此，我猜what...the的主pod标签设置不正确。

现在您可以看到选择器标签显示在这里：

> aws-vault exec nonlive-build -- kubectl get po -n cicd --show-labels
NAME                                  READY   STATUS    RESTARTS   AGE    LABELS
jenkins-6597db4979-khxls              2/2     Running   0          4m8s   app.kubernetes.io/component=main,app.kubernetes.io/instance=jenkins

为此，将新条目添加到值文件中：

main:
  metadata:
    labels:
      app.kubernetes.io/name: jenkins
      app.kubernetes.io/component: main

并且相应地更新了Helm _helpers.tpl模板：

{{- define "jenkins.selectorLabels" -}}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- if .Values.main.metadata.labels }}
{{- range $k, $v := .Values.main.metadata.labels }}
{{ $k }}: {{ $v }}
{{- end }}
{{- end }}
{{- end }}