state_replicaset/state_replicaset.go98发出http请求时出错: Get kube-state-metrics:8080/metrics: lookup kube-state-metrics on IP:53没有这样的主机

Question

我们正在尝试在台风kubernetes集群上启动metricbeat。但在启动后，它不能获得一些pod特定的事件，如重新启动等，因为以下原因

相应的metricbeat.yaml代码段

# State metrics from kube-state-metrics service:
    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
        - state_service
        - state_persistentvolume
        - state_persistentvolumeclaim
        - state_storageclass
        # Uncomment this to get k8s events:
        #- event  period: 10s
      hosts: ["kube-state-metrics:8080"]

我们所面临的错误

2020-07-01T10:31:02.486Z        ERROR   [kubernetes.state_statefulset]  state_statefulset/state_statefulset.go:97       error making http request: Get http://kube-state-metrics:8080/metrics: lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:02.611Z        WARN    [transport]     transport/tcp.go:52     DNS lookup failure "kube-state-metrics": lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:02.611Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.state_node: error doing HTTP request to fetch 'state_node' Metricset data: error making http request: Get http://kube-state-metrics:8080/metrics: lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:03.313Z        ERROR   process_summary/process_summary.go:102  Unknown or unexpected state <P> for process with pid 19
2020-07-01T10:31:03.313Z        ERROR   process_summary/process_summary.go:102  Unknown or unexpected state <P> for process with pid 20

我可以添加一些其他信息，这是必需的。

Answer 1

确保在集群中的kube-system名称空间中部署了Kube-State-Metrics才能正常工作。默认情况下，Metricbeat不会提供此功能。

请参考this了解详细的部署说明。

Answer 2

如果您的kube-state-metrics部署到另一个命名空间，Kubernetes将无法解析该名称。例如，我们在monitoring名称空间中部署了kube-state-metrics：

$ kubectl get pods -A | grep kube-state-metrics
monitoring             kube-state-metrics-765c7c7f95-v7mmp                             3/3     Running   17         10d

您可以将hosts选项设置为全名，包括名称空间，如下所示：

    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
        - state_service
        - state_persistentvolume
        - state_persistentvolumeclaim
        - state_storageclass
      hosts: ["kube-state-metrics.<your_namespace>:8080"]