分区网络终结点组在通过GKE提供托管证书后不健康

Question

在GCP中配置带有托管证书的入口后，我发现Zonal network endpoint group不健康

# kubernetes/backstage.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backstage
  namespace: backstage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backstage
  template:
    metadata:
      labels:
        app: backstage
    spec:
      containers:
        - name: backstage
          image: australia-southeast1-docker.pkg.dev/acme-dev-tooling/acme-docker/backstage:prd-v.0.35
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 7007
          envFrom:
            - secretRef:
                name: postgres-secrets
            - secretRef:
                name: backstage-secrets
---
apiVersion: v1
kind: Service
metadata:
  name: backstage
  namespace: backstage
  annotations:
    cloud.google.com/backend-config: '{"default": "backstage-ingress-backendconfig"}'
spec:
  selector:
    app: backstage
  ports:
    - name: http
      protocol: TCP
      port: 80
  type: NodePort
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: backstage-ingress-backendconfig
spec:
  healthCheck:
    checkIntervalSec: 15
    type: HTTP
    requestPath: /
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: tools-managed-cert-backstage
  namespace: backstage
spec:
  domains:
    - tools.backstage.acme-uat.com
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backstage-ingress
  namespace: backstage
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "tools-backstage-external-ip"
    networking.gke.io/managed-certificates: tools-managed-cert-backstage
    kubernetes.io/ingress.class: "gce"
spec:
  defaultBackend:
    service:
      name: backstage
      port:
        number: 80
---
apiVersion: v1
kind: Namespace
metadata:
  name: backstage

GCP提供了一个L7 https负载平衡器，由于分区健康端点连接，它无法访问GKE集群。

入口写着：

所有后端都处于不健康的状态。

我遗漏了什么吗？GKE入口是否配置防火墙？我看过这些规则，有关于130.211.0.0/22，35.191.0.0/16的规则，这是健康检查地址。

logs/compute.googleapis.com%2Fhealthchecks不产生探针结果。尽管启用了日志记录。

任何帮助都将不胜感激。

UPDATE -每条评论都是固定的，下面的内容不起作用

kind: Service
metadata:
  name: argocd-server
  namespace: argocd
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    argocd.argoproj.io/instance: argocd
    helm.sh/chart: argo-cd-3.29.5
  annotations:
    cloud.google.com/backend-config: '{"default": "argocd-ingress-backendconfig"}'
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/neg-status: >-
      {"network_endpoint_groups":{"80":"k8s1-20a3d3ad-argocd-argocd-server-80-c2ec22fa"},"zones":["australia-southeast1-a"]}
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/backend-config":"{\"default\":
      \"argocd-ingress-backendconfig\"}","cloud.google.com/neg":"{\"ingress\":
      true}"},"labels":{"app.kubernetes.io/component":"server","app.kubernetes.io/instance":"argocd","app.kubernetes.io/managed-by":"pulumi","app.kubernetes.io/name":"argocd-server","app.kubernetes.io/part-of":"argocd","argocd.argoproj.io/instance":"argocd","helm.sh/chart":"argo-cd-3.29.5"},"name":"argocd-server","namespace":"argocd"},"spec":{"ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":8080},{"name":"https","port":443,"protocol":"TCP","targetPort":"http"}],"selector":{"app.kubernetes.io/instance":"argocd","app.kubernetes.io/name":"argocd-server"},"type":"ClusterIP"}}
status:
  loadBalancer: {}
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: https
      protocol: TCP
      port: 443
      targetPort: http
  selector:
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/name: argocd-server
  clusterIP: 10.184.10.20
  clusterIPs:
    - 10.184.10.20
  type: ClusterIP
  sessionAffinity: None
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
  namespace: argocd
  uid: fee5f91c-b431-4b8c-ab10-64daa02ec729
  resourceVersion: '108355'
  generation: 3
  creationTimestamp: '2022-01-20T00:06:05Z'
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.2.2
    helm.sh/chart: argo-cd-3.30.1
  annotations:
    deployment.kubernetes.io/revision: '3'
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"labels":{"app.kubernetes.io/component":"server","app.kubernetes.io/instance":"argocd","app.kubernetes.io/managed-by":"pulumi","app.kubernetes.io/name":"argocd-server","app.kubernetes.io/part-of":"argocd","app.kubernetes.io/version":"v2.2.2","helm.sh/chart":"argo-cd-3.30.1"},"name":"argocd-server","namespace":"argocd"},"spec":{"replicas":1,"revisionHistoryLimit":5,"selector":{"matchLabels":{"app.kubernetes.io/instance":"argocd","app.kubernetes.io/name":"argocd-server"}},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"server","app.kubernetes.io/instance":"argocd","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"argocd-server","app.kubernetes.io/part-of":"argocd","app.kubernetes.io/version":"v2.2.2","helm.sh/chart":"argo-cd-3.30.1"}},"spec":{"containers":[{"command":["argocd-server","--staticassets","/shared/app","--repo-server","argocd-repo-server:8081","--dex-server","http://argocd-dex-server:5556","--logformat","text","--loglevel","info","--redis","argocd-redis:6379"],"image":"quay.io/argoproj/argocd:v2.2.2","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":8080},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"server","ports":[{"containerPort":8080,"name":"server","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":8080},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{},"volumeMounts":[{"mountPath":"/app/config/ssh","name":"ssh-known-hosts"},{"mountPath":"/app/config/server/tls","name":"argocd-repo-server-tls"},{"mountPath":"/home/argocd","name":"plugins-home"},{"mountPath":"/tmp","name":"tmp-dir"}]}],"serviceAccountName":"argocd-server","volumes":[{"emptyDir":{},"name":"static-files"},{"emptyDir":{},"name":"tmp-dir"},{"configMap":{"name":"argocd-ssh-known-hosts-cm"},"name":"ssh-known-hosts"},{"name":"argocd-repo-server-tls","secret":{"items":[{"key":"tls.crt","path":"tls.crt"},{"key":"tls.key","path":"tls.key"},{"key":"ca.crt","path":"ca.crt"}],"optional":true,"secretName":"argocd-repo-server-tls"}},{"emptyDir":{},"name":"plugins-home"}]}}}}
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: argocd
      app.kubernetes.io/name: argocd-server
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: server
        app.kubernetes.io/instance: argocd
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: argocd-server
        app.kubernetes.io/part-of: argocd
        app.kubernetes.io/version: v2.2.2
        helm.sh/chart: argo-cd-3.30.1
      annotations:
        kubectl.kubernetes.io/restartedAt: '2022-01-20T15:44:27+11:00'
    spec:
      volumes:
        - name: static-files
          emptyDir: {}
        - name: tmp-dir
          emptyDir: {}
        - name: ssh-known-hosts
          configMap:
            name: argocd-ssh-known-hosts-cm
            defaultMode: 420
        - name: argocd-repo-server-tls
          secret:
            secretName: argocd-repo-server-tls
            items:
              - key: tls.crt
                path: tls.crt
              - key: tls.key
                path: tls.key
              - key: ca.crt
                path: ca.crt
            defaultMode: 420
            optional: true
        - name: plugins-home
          emptyDir: {}
      containers:
        - name: server
          image: quay.io/argoproj/argocd:v2.2.2
          command:
            - argocd-server
            - '--staticassets'
            - /shared/app
            - '--repo-server'
            - argocd-repo-server:8081
            - '--dex-server'
            - http://argocd-dex-server:5556
            - '--logformat'
            - text
            - '--loglevel'
            - info
            - '--redis'
            - argocd-redis:6379
          ports:
            - name: server
              containerPort: 8080
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: ssh-known-hosts
              mountPath: /app/config/ssh
            - name: argocd-repo-server-tls
              mountPath: /app/config/server/tls
            - name: plugins-home
              mountPath: /home/argocd
            - name: tmp-dir
              mountPath: /tmp
          livenessProbe:
            httpGet:
              path: /
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: argocd-server
      serviceAccount: argocd-server
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 5
  progressDeadlineSeconds: 600

干杯

Answer 1

# Here is workaround for Google Cloud with ArgoCD v2.5.2
# cloudflare-key.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-key
  namespace: cert-manager
type: Opaque
stringData:
  key: xxxxxxxxxxxxxxxx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: zia@mydomain.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - selector: {}
        dns01:
          cloudflare:
            email: zia@mydomain.com
            apiKeySecretRef:
              name: cloudflare-key
              key: key
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/component: server
  annotations:
    cloud.google.com/neg: '{"ingress": true, "exposed_ports": {"8080":{}}}'
    beta.cloud.google.com/backend-config: '{"default": "argocd-backend-config"}'
  name: argocd-server
spec:
  ports:
  - name: http8080
    protocol: TCP
    port: 8080
    targetPort: 8080
  - name: http
    protocol: TCP
    port: 80
    targetPort: 8080
  - name: https
    protocol: TCP
    port: 443
    targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server
---
#backendconfig.yaml
kind: BackendConfig
metadata:
  name: argocd-backend-config
  namespace: argocd
spec:
  healthCheck:
    checkIntervalSec: 30
    timeoutSec: 10
    healthyThreshold: 1
    unhealthyThreshold: 5
    type: HTTP
    requestPath: /healthz
    port: 8080
---
# FrontendConfig.yaml
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: argocd-frontend-config
  namespace: argocd
spec:
  redirectToHttps:
    enabled: true
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: gce
    cert-manager.io/cluster-issuer: letsencrypt-staging
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.global-static-ip-name: "argocd-dev"
    networking.gke.io/v1beta1.FrontendConfig: argocd-frontend-config
spec:
  rules:
  - host: argocd-dev.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: argocd-server
            port: 
              name: http
        path: "/"
        pathType: Prefix
  tls:
  - hosts:
      - argocd-dev.mydomain.com
    secretName: argocd-secret #don't change, this is provided by ArgoCD