EKS外部-Dns IAM故障排除

Question

问题

我正在尝试排除以下信息。

time="<timestamp>" level=error msg="records retrieval failed: failed to list hosted zones: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: <uuid>"

我从运行kubectl logs external-dns-xxxxxxxxxx-xxxxx中得到的

我的问题

我想弄清楚..。

1. 从哪里生成这条消息？我不明白它是来自我的服务，服务帐户，集群角色，集群绑定，吊舱，还是别的什么。如有任何澄清或与有用解释的联系，将不胜感激。(根据k8s文档，我现在的猜测是来自荚，但我仍然不确定如何跟踪它以确认)
2. 为什么我已经明确指定的IAM权限不是由我的外部dns假设的？任何关于我的外部dns荚如何尝试执行其许可假设或执行其任务的流程的任何解释都将不胜感激！

我的目标

我对K8s非常陌生，并且正在尝试部署EKS集群w/外部-dns，以便自动管理我的Route53记录。

到目前为止我尝试过的

我在扩展IAM权限方面做了很多工作，并尽可能广泛地打开这些权限。我在定义eks.amazonaws.com/role-arn

• I've的所有资源中显式添加了注释，尝试将external-dns部署从kube-system迁移到default命名空间，因为这是针对具有相同错误消息的GitHub问题推荐的。

部署细节

我使用Terraform部署我的大部分EKS集群、节点组、OIDC和& Helm。

现在，我选择了只分享部署的结果，而不是倾诉，以尽量减少这个问题的规模。如果你想看那些吐露的话，只要问一问，我就会分享我所拥有的一切。

Kubectl描述

kubectl describe service external-dns

Name:              external-dns
Namespace:         default
Labels:            app.kubernetes.io/instance=external-dns
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=external-dns
                   helm.sh/chart=external-dns-6.9.0
Annotations:       eks.amazonaws.com/role-arn: arn:aws:iam::<user-id>:role/AllowExternalDNSUpdates
                   meta.helm.sh/release-name: external-dns
                   meta.helm.sh/release-namespace: default
Selector:          app.kubernetes.io/instance=external-dns,app.kubernetes.io/name=external-dns
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.20.233.113
IPs:               172.20.233.113
Port:              http  7979/TCP
TargetPort:        http/TCP
Endpoints:         10.12.13.93:7979
Session Affinity:  None
Events:            <none>

kubectl describe serviceaccount external-dns

Name:                external-dns
Namespace:           default
Labels:              app.kubernetes.io/managed-by=Helm
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<user-id>:role/AllowExternalDNSUpdates
                     meta.helm.sh/release-name: external-dns
                     meta.helm.sh/release-namespace: default
Image pull secrets:  <none>
Mountable secrets:   external-dns-token-twgpb
Tokens:              external-dns-token-twgpb
Events:              <none>

kubectl describe clusterrole external-dns

Name:         external-dns
Labels:       <none>
Annotations:  eks.amazonaws.com/role-arn: arn:aws:iam::<user-id>:role/AllowExternalDNSUpdates
PolicyRule:
  Resources                     Non-Resource URLs  Resource Names  Verbs
  ---------                     -----------------  --------------  -----
  endpoints                     []                 []              [get watch list]
  nodes                         []                 []              [get watch list]
  pods                          []                 []              [get watch list]
  services                      []                 []              [get watch list]
  ingresses.extensions          []                 []              [get watch list]
  gateways.networking.istio.io  []                 []              [get watch list]
  ingresses.networking.k8s.io   []                 []              [get watch list]

kubectl describe clusterrolebindings.rbac.authorization.k8s.io external-dns

Name:         external-dns
Labels:       <none>
Annotations:  eks.amazonaws.com/role-arn: arn:aws:iam::<user-id>:role/AllowExternalDNSUpdates
Role:
  Kind:  ClusterRole
  Name:  external-dns
Subjects:
  Kind            Name          Namespace
  ----            ----          ---------
  ServiceAccount  external-dns  default

kubectl describe ingress -n kube-system

Name:             aws-lb-ctrlr
Labels:           <none>
Namespace:        kube-system
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /*   aws-load-balancer-controller:80 (<error: endpoints "aws-load-balancer-controller" not found>)
Annotations:  alb.ingress.kubernetes.io/inbound-cidrs: 0.0.0.0/0
              alb.ingress.kubernetes.io/listen-ports: [{'HTTP': 80}]
              alb.ingress.kubernetes.io/scheme: internet-facing
              external-dns.alpha.kubernetes.io/hostname: <my-domain.tld>
              kubernetes.io/ingress.class: alb
Events:       <none>

kubectl describe pod

Name:             external-dns-xxxxxxxxxx-xxxxx
Namespace:        default
Priority:         0
Service Account:  external-dns
Node:             ip-10-12-13-107.ec2.internal/10.12.13.107
Start Time:       Tue, 20 Sep 2022 10:48:06 -0400
Labels:           app.kubernetes.io/instance=external-dns
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=external-dns
                  helm.sh/chart=external-dns-6.9.0
                  pod-template-hash=xxxxxxxxxx
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               10.12.13.93
IPs:
  IP:           10.12.13.93
Controlled By:  ReplicaSet/external-dns-xxxxxxxxxx
Containers:
  external-dns:
    Container ID:  docker://5b49f49f7b9c0be8cb00835f117eedccaff3d5bb4ebfecb4bc6af771d2b3d336
    Image:         docker.io/bitnami/external-dns:0.12.2-debian-11-r14
    Image ID:      docker-pullable://bitnami/external-dns@sha256:195dec0f60c9137952ea0604623c7eb001ece4142916bdfb0cc79f5d9cdc4b62
    Port:          7979/TCP
    Host Port:     0/TCP
    Args:
      --metrics-address=:7979
      --log-level=debug
      --log-format=text
      --domain-filter=<my-domain.tld>
      --policy=sync
      --provider=aws
      --registry=txt
      --interval=1m
      --txt-owner-id=<hosted-zone-id>
      --source=service
      --source=ingress
      --aws-api-retries=3
      --aws-zone-type=public
      --aws-assume-role=arn:aws:iam::<user-id>:role/AllowExternalDNSUpdates
      --aws-batch-change-size=1000
    State:          Running
      Started:      Tue, 20 Sep 2022 10:48:13 -0400
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/healthz delay=10s timeout=5s period=10s #success=1 #failure=2
    Readiness:      http-get http://:http/healthz delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      AWS_DEFAULT_REGION:           us-east-1
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_ROLE_ARN:                 arn:aws:iam::<user-id>:role/AllowExternalDNSUpdates
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d82r7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kube-api-access-d82r7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m44s  default-scheduler  Successfully assigned default/external-dns-xxxxxxxxxx-xxxxx to ip-10-12-13-107.ec2.internal
  Normal  Pulling    3m43s  kubelet            Pulling image "docker.io/bitnami/external-dns:0.12.2-debian-11-r14"
  Normal  Pulled     3m40s  kubelet            Successfully pulled image "docker.io/bitnami/external-dns:0.12.2-debian-11-r14" in 3.588418583s
  Normal  Created    3m38s  kubelet            Created container external-dns
  Normal  Started    3m37s  kubelet            Started container external-dns

kubectl describe deployments.apps

Name:                   external-dns
Namespace:              default
CreationTimestamp:      Tue, 20 Sep 2022 10:48:06 -0400
Labels:                 app.kubernetes.io/instance=external-dns
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=external-dns
                        helm.sh/chart=external-dns-6.9.0
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: external-dns
                        meta.helm.sh/release-namespace: default
Selector:               app.kubernetes.io/instance=external-dns,app.kubernetes.io/name=external-dns
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=external-dns
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=external-dns
                    helm.sh/chart=external-dns-6.9.0
  Service Account:  external-dns
  Containers:
   external-dns:
    Image:      docker.io/bitnami/external-dns:0.12.2-debian-11-r14
    Port:       7979/TCP
    Host Port:  0/TCP
    Args:
      --metrics-address=:7979
      --log-level=debug
      --log-format=text
      --domain-filter=<my-domain.tld>
      --policy=sync
      --provider=aws
      --registry=txt
      --interval=1m
      --txt-owner-id=<hosted-zone-id>
      --source=service
      --source=ingress
      --aws-api-retries=3
      --aws-zone-type=public
      --aws-assume-role=arn:aws:iam::<user-id>:role/AllowExternalDNSUpdates
      --aws-batch-change-size=1000
    Liveness:   http-get http://:http/healthz delay=10s timeout=5s period=10s #success=1 #failure=2
    Readiness:  http-get http://:http/healthz delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      AWS_DEFAULT_REGION:  us-east-1
    Mounts:                <none>
  Volumes:                 <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   external-dns-xxxxxxxxxx (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  9m30s  deployment-controller  Scaled up replica set external-dns-xxxxxxxxxx to 1

AWS IAM (AllowExternalDNSUpdates)

IAM作用(信任关系)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<userid>:oidc-provider/oidc.eks.region-code.amazonaws.com/id/<oidc-id>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity"
        }
    ]
}

IAM策略(许可)

{
    "Statement": [
        {
            "Action": "route53:ChangeResourceRecordSets",
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/*",
            "Sid": ""
        },
        {
            "Action": [
                "route53:ListResourceRecordSets",
                "route53:ListHostedZones"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

Answer 1

回答

所以基本上是两件事

1. (Credit @Jordanm，在评论中)信任关系是不正确的，我编辑了这篇文章来修正它，并重新运行我的信任。然后我的问题变成了records retrieval failed: failed to list hosted zones: AccessDenied: User: arn:aws:sts::<userid>:assumed-role/AllowExternalDNSUpdates/1663776911448118272 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<userid>:role/AllowExternalDNSUpdates\n\tstatus
2. Because -我有一个额外的错误，我必须回去修复我的terraform配置，并删除“假设角色”设置。基本上，如果您有第二个错误，“假设-角色试图承担-角色”，那么您只是两次承担这个角色。