如何将scrape_configs文件添加到Prometheus

Question

我使用牧场主监视Helm图表在cattle-monitoring-system命名空间中部署了Prometheus、Alertmanager、Grafana等。有关文档，请参阅这里和这里。这个命名空间中的所有内容都运行正常。

现在，我试图使用清单文件在另一个名为indigo的名称空间中部署一个额外的Prometheus实例。不幸的是，配置scrape_configs文件不会加载。我是通过运行命令kubectl exec -it prometheus-indigo-0 -c prometheus -n indigo -- cat /etc/prometheus/config_out/prometheus.env.yaml来检查的。在其他Prometheus实例中运行相同的命令将返回所有scrape_configs配置。

我的deployment.yml文件如下所示：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monitoring-rancher-monitor-prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: indigo
  namespace: $NAMESPACE
spec:
  serviceAccountName: prometheus
  additionalScrapeConfigs:
    name: prometheus-scrape-configs-secret
    key: prometheus-scrape-configs.yml
  resources:
    requests:
      memory: 400Mi
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-indigo
  namespace: $NAMESPACE
spec:
  type: NodePort
  ports:
  - name: web
    nodePort: 30900
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    prometheus: indigo

prometheus-scrape-configs.yml文件如下所示：

global:
  evaluation_interval: 30s
  scrape_interval: 30s
scrape_configs:
- job_name: blackbox # To get metrics about the exporter itself
  metrics_path: /metrics
  static_configs:
    - targets:
      - .....

- job_name: blackbox-http # To get metrics about the exporter’s targets
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - .....
      labels:
        env: elise
    - targets:
      - .....
      labels:
        env: osb
    - targets:
      - .....
      labels:
        env: itp
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: .....

最后，在prometheus-scrape-configs-secret中创建Makefile

include ../../../common.mk
include ../../environments/$(ENVIRONMENT).mk

NAMESPACE ?= indigo

.PHONY: deploy
deploy: \
    init \
    deploy-monitoring

.PHONY: init
init:

.PHONY: deploy-monitoring
deploy-monitoring:
    kubectl create secret generic prometheus-scrape-configs-secret \
    -n $(NAMESPACE) --from-file=prometheus-scrape-configs.yml

    envsubst < $(ENVIRONMENT)-deployment.yml | kubectl apply -f -

.PHONY: delete
delete: delete-monitoring

.PHONY: delete-monitoring
delete-monitoring:
    kubectl delete secret prometheus-scrape-configs-secret -n $(NAMESPACE)
    envsubst < $(ENVIRONMENT)-deployment.yml | kubectl delete -f -

那么，有人知道为什么scrape_configs文件无法正确加载吗？

编辑：

在运行命令kubectl describe pod prometheus-indigo-0 -n indigo之后，我注意到下面出现了这个错误。我还注意到，在部署这个Prometheus实例之后，它将得到一个错误，立即重新启动，然后运行。

State:       Running
  Started:   Tue, 14 Jun 2022 18:01:23 +0200
Last State:  Terminated
  Reason:    Error
  Message:   ts=2022-06-14T16:01:19.743Z caller=main.go:450 level=error msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"

Answer 1

多亏了这些文档我设法解决了这个问题。我忘记创建一个通过标签引用blackbox-exporter的blackbox-exporter。最后，您需要使用serviceMonitorSelector将其链接到Kubernetes Prometheus对象中。您可以在这里查找更多信息。如何配置serviceMonitorSelector