通过Terraform和Helm添加和安装Bitnami外部DNS后NMI吊舱中的错误-没有找到AzureIdentityBinding

Question

我正在努力获得azureIdentity的ExternalDNS绑定，并将ExternalDNS条目输入到我们的区域中。

键错误: I0423 19:27:52.830107 1 mic.go:610]没有找到符合选择器:外部-dns默认/外部-dns-84dcc5f68c-cl5h5的AzureIdentityBinding。它也将被忽略，没有创建azureAssignedIdentity，因为没有匹配的荚和选择器/aadpodidbinding。

我正在使用Terraform，Helm，Azure，Azure AKS，VSCODE构建IaaC，到目前为止，有三个Kubernetes附加组件-add标识，应用程序-网关-kubernetes-入口和Bitnami外部-dns。

由于标识没有被绑定，所以不会创建azureAssignedIdentity，而且ExternalDNS无法将记录放入我们的DNS区域。

名字和装订似乎都是正确的。我尝试在Terraform kubectl_manifest提供程序中传递用于Bitnami ExternalDNS的Helm安装的ExternalDNS。我试过压制ExternalDNS名称和标签上的后缀。我尝试编辑集群本身上的Helm和Kubernetes YAML，以尝试强制绑定。我尝试使用AKS用户管理标识，该标识用于AAD标识，位于集群的节点池资源组中。我已经尝试过让Bitnami ExternalDNS配置和添加一个azure.json文件，并且在添加和安装ExternalDNS之前我也手动完成了。我尝试将托管标识分配给AKS集群的VMSS。

谢谢!

JBP

​

PS C:\Workspace\tf\HelmOne> kubectl logs pod/external-dns-84dcc5f68c-542mv
: Refresh request failed. Status Code = '404'. Response body: getting assigned identities for pod default/external-dns-84dcc5f68c-542mv in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors\n"
time="2021-04-24T19:57:30Z" level=debug msg="Retrieving Azure DNS zones for resource group: one-hi-sso-dnsrg-tf."
time="2021-04-24T20:06:02Z" level=error msg="azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-dnsrg-tf/providers/Microsoft.Network/dnsZones?api-version=2018-05-01: StatusCode=404 -- Original Error: adal: Refresh request failed. Status Code = '404'. Response body: getting assigned identities for pod default/external-dns-84dcc5f68c-542mv in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors\n"
time="2021-04-24T20:06:02Z" level=debug msg="Retrieving Azure DNS zones for resource group: one-hi-sso-dnsrg-tf."



PS C:\Workspace\tf\HelmOne> kubectl logs pod/aad-pod-identity-nmi-vtmwm
I0424 20:07:22.400942       1 server.go:196] status (404) took 80007557875 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.8.7
E0424 20:08:44.427353       1 server.go:375] failed to get matching identities for pod: default/external-dns-84dcc5f68c-542mv, error: getting assigned identities for pod default/external-dns-84dcc5f68c-542mv in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0424 20:08:44.427400       1 server.go:196] status (404) took 80025612263 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.8.7



PS C:\Workspace\TF\HelmOne>  kubectl logs pod/aad-pod-identity-mic-86944f67b8-k4hds
I0422 21:05:11.298958       1 main.go:114] starting mic process. Version: v1.7.5. Build date: 2021-04-02-21:14
W0422 21:05:11.299031       1 main.go:119] --kubeconfig not passed will use InClusterConfig
I0422 21:05:11.299038       1 main.go:136] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0422 21:05:11.299205       1 main.go:144] running MIC in namespaced mode: false
I0422 21:05:11.299223       1 main.go:148] client QPS set to: 5. Burst to: 5
I0422 21:05:11.299243       1 mic.go:139] starting to create the pod identity client. Version: v1.7.5. Build date: 2021-04-02-21:14
I0422 21:05:11.318835       1 mic.go:145] Kubernetes server version: v1.18.14
I0422 21:05:11.319465       1 cloudprovider.go:122] MIC using user assigned identity: c380##### REDACTED #####814b for authentication.
I0422 21:05:11.392322       1 probes.go:41] initialized health probe on port 8080
I0422 21:05:11.392351       1 probes.go:44] started health probe
I0422 21:05:11.392458       1 metrics.go:341] registered views for metric
I0422 21:05:11.392544       1 prometheus_exporter.go:21] starting Prometheus exporter
I0422 21:05:11.392561       1 metrics.go:347] registered and exported metrics on port 8888
I0422 21:05:11.392568       1 mic.go:244] initiating MIC Leader election
I0422 21:05:11.393053       1 leaderelection.go:243] attempting to acquire leader lease  default/aad-pod-identity-mic...
E0423 01:47:52.730839       1 leaderelection.go:325] error retrieving resource lock default/aad-pod-identity-mic: etcdserver: request timed out



resource "helm_release" "external-dns" {
    name          = "external-dns"
    repository    = "https://charts.bitnami.com/bitnami"
    chart         = "external-dns"
    namespace     = "default"
    version       = "4.0.0"
  
  set {
      name  = "azure.cloud"
      value = "AzurePublicCloud"
  }
  
  #MyDnsResourceGroup
  set {
      name  = "azure.resourceGroup"
      value = data.azurerm_resource_group.dnsrg.name 
  }
  
  set {
      name  = "azure.tenantId"
      value = data.azurerm_subscription.currenttenantid.tenant_id
  }
  
  set {
      name  = "azure.subscriptionId"
      value = data.azurerm_subscription.currentSubscription.subscription_id
  }
  
  set {
      name    = "azure.userAssignedIdentityID"
      value   = azurerm_user_assigned_identity.external-dns-mi-tf.client_id
  }
  
  #Verbosity of the logs (options: panic, debug, info, warning, error, fatal, trace)
  set {
      name  = "logLevel"
      value = "trace" 
  }
  
  set {
      name  = "sources"
      value = "{service,ingress}"
  }
  
  set {
      name  = "domainFilters"
      value = "{${var.child_domain_prefix}.${lower(var.parent_domain)}}"
  }
  
  #DNS provider where the DNS records will be created (mandatory) (options: aws, azure, google, ...)
  set {
      name  = "provider"
      value = "azure"
  }
  
  #podLabels: {aadpodidbinding: <selector>} # selector you defined above in AzureIdentityBinding 
  set {
      name  = "podLabels.aadpodidbinding"
      value = "external-dns"
  }
  
  set {
      name = "azure.useManagedIdentityExtension"
      value = true
  }
}  



resource "helm_release" "aad-pod-identity" {
    name       = "aad-pod-identity"
    repository = "https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts"
    chart      = "aad-pod-identity"
  }


  resource "helm_release" "ingress-azure" {
    name          = "ingress-azure"
    repository    = "https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/"
    chart         = "ingress-azure"
    namespace     = "default"
    version       = "1.4.0"
  
  set {
      name        = "debug"
      value       = "true"
  }
  
  set {
      name        = "appgw.name"
      value       = data.azurerm_application_gateway.appgwpub.name
  }
  
  set {
      name        = "appgw.resourceGroup"
      value       = data.azurerm_resource_group.appgwpubrg.name
  }
  
  set {
      name        = "appgw.subscriptionId"
      value       = data.azurerm_subscription.currentSubscription.subscription_id
  }
  
  set {
      name        = "appgw.usePrivateIP"
      value       = "false"
  }
  
  set {
      name        = "armAuth.identityClientID"
      value       = azurerm_user_assigned_identity.agic-mi-tf.client_id
  }
  
  set {
      name        = "armAuth.identityResourceID"
      value       =  azurerm_user_assigned_identity.agic-mi-tf.id
  }
  
  set {
      name        = "armAuth.type"
      value       = "aadPodIdentity"
  }
  
  set {
      name        = "rbac.enabled"
      value       = "true"
  }
  
  set {
      name        = "verbosityLevel"
      value       = "5"
  }
  
  set {
      name        = "appgw.environment"
      value       = "AZUREPUBLICCLOUD"
  }
  
  set {
      name        = "metadata.name"
      value       = "ingress-azure"
  }
  
  }



PS C:\Workspace\tf\HelmOne> kubectl get azureassignedidentities
NAME                                                   AGE
ingress-azure-68c97fd496-qbptf-default-ingress-azure   23h


PS C:\Workspace\tf\HelmOne> kubectl get azureidentity
NAME                            AGE
ingress-azure                   23h
one-hi-sso-agic-mi-tf           23h
one-hi-sso-external-dns-mi-tf   23h


PS C:\Workspace\tf\HelmOne> kubectl edit azureidentity one-hi-sso-external-dns-mi-tf
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentity","metadata":{"annotations":{},"name":"one-hi-sso-external-dns-mi-tf","namespace":"default"},"spec":{"clientID":"f58e7c55-REDACTED-a6e358e53912","resourceID":"/subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-kuberg-tf/providers/Microsoft.ManagedIdentity/userAssignedIdentities/one-hi-sso-external-dns-mi-tf","type":0}}
  creationTimestamp: "2021-04-22T20:44:42Z"
  generation: 2
  name: one-hi-sso-external-dns-mi-tf
  namespace: default
  resourceVersion: "432055"
  selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/default/azureidentities/one-hi-sso-external-dns-mi-tf
  uid: f8e22fd9-REDACTED-6cdead0d7e22
spec:
  clientID: f58e7c55-REDACTED-a6e358e53912
  resourceID: /subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-kuberg-tf/providers/Microsoft.ManagedIdentity/userAssignedIdentities/one-hi-sso-external-dns-mi-tf
  type: 0


PS C:\Workspace\tf\HelmOne> kubectl edit azureidentitybinding external-dns-mi-binding
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentityBinding","metadata":{"annotations":{},"name":"external-dns-mi-binding","namespace":"default"},"spec":{"AzureIdentity":"one-hi-sso-external-dns-mi-tf","Selector":"external-dns"}}
  creationTimestamp: "2021-04-22T20:44:42Z"
  generation: 1
  name: external-dns-mi-binding
  namespace: default
  resourceVersion: "221101"
  selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/default/azureidentitybindings/external-dns-mi-binding
  uid: f39e7418-e896-4b8e-b596-035cf4b66252
spec:
  AzureIdentity: one-hi-sso-external-dns-mi-tf
  Selector: external-dns



  resource "kubectl_manifest" "one-hi-sso-external-dns-mi-tf" {
    yaml_body = <<YAML
apiVersion: "aadpodidentity.k8s.io/v1"  
kind: AzureIdentity
metadata:
  name: one-hi-sso-external-dns-mi-tf
  namespace: default
spec:
  type: 0
  resourceID: /subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-kuberg-tf/providers/Microsoft.ManagedIdentity/userAssignedIdentities/one-hi-sso-external-dns-mi-tf
  clientID: f58e7c55-REDACTED-a6e358e53912
  YAML
}


resource "kubectl_manifest" "external-dns-mi-binding" {
    yaml_body = <<YAML
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: external-dns-mi-binding
spec:
  AzureIdentity: one-hi-sso-external-dns-mi-tf
  Selector: external-dns
YAML
}

​

Answer 1

我使用的托管标识没有添加到虚拟机缩放集VMSS中。一旦我添加了它，绑定就可以工作了，并且创建了azureAssignedIdentity。

此外，我将AzureIdentity和AzureIdentity YAML中的选择行从大写的首字母转换为小写的首字母。

更正: azureIdentity:选择器：