GKE中弹性搜索的StackDriver监测

Question

我需要监视安装在k8s集群之上的Elasticsearch(2.4)。我有2个客户端，3个主服务器和几个数据节点在荚中运行。按照Stackdriver的“如何”和post "Can I run Google Monitoring Agent inside a Kubernetes Pod?"，我在它自己的Pod中部署了一个代理。毕竟，突然之间，StackDriver中就没有弹性搜索指标了。唯一的零。

​

​

任何建议都是受欢迎的。

这是我的配置：

弹性服务：

$kubectl describe svc elasticsearch
Name:           elasticsearch
Namespace:      default
Labels:         component=elasticsearch
            role=client
Selector:       component=elasticsearch,role=client
Type:           NodePort
IP:         <IP>
Port:           http    9200/TCP
NodePort:       http    <PORT>/TCP
Endpoints:      <IP>:9200,<IP>:9200
Session Affinity:   None
No events.

堆栈驱动程序部署：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: stackagent
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: monitoring
        role: stackdriver-agent
    spec:
      containers:
      - name: hslab-data-agent
        image: StackDriverAgent:version1

StackDriverAgent:版本1 Docker：

FROM ubuntu

WORKDIR /stackdriver

RUN apt-get update
RUN apt-get install curl lsb-release libyajl2 -y
RUN apt-get clean

COPY ./stackdriver/run.sh run.sh
COPY ./stackdriver/elasticsearch.conf elasticsearch.conf

RUN chmod 755 ./run.sh
CMD ["./run.sh"]

run.sh：

#!/bin/bash

curl -O https://repo.stackdriver.com/stack-install.sh

chmod 755 stack-install.sh
bash stack-install.sh --write-gcm

cp ./elasticsearch.conf /opt/stackdriver/collectd/etc/collectd.d/

service stackdriver-agent restart

while true; do
    sleep 60
    agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null)

    ps -p $agent_pid > /dev/null 2>&1
    if [ $? != 0 ]; then
        echo "Stackdriver agent pid not found!"
        break;
    fi
done

elasticsearch.conf：

取自https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/elasticsearch.conf

# This is the monitoring configuration for Elasticsearch 1.0.x and later.
# Look for ELASTICSEARCH_HOST and ELASTICSEARCH_PORT to adjust your configuration file.
LoadPlugin curl_json
<Plugin "curl_json">
    # When using non-standard Elasticsearch configurations, replace the below with
    #<URL "http://ELASTICSEARCH_HOST:ELASTICSEARCH_PORT/_nodes/_local/stats/">
    # PREVIOUSE LINE
    # <URL "http://localhost:9200/_nodes/_local/stats/"> 
    <URL "http://elasticsearch:9200/_nodes/_local/stats/">
        Instance "elasticsearch"
 ....

运行状态：

NAME                                      READY     STATUS    RESTARTS   AGE
esclient-4231471109-bd4tb                 1/1       Running   0          23h
esclient-4231471109-k5pnw                 1/1       Running   0          23h
esdata-1-2524076994-898r0                 1/1       Running   0          23h
esdata-2-2426789420-zhz7j                 1/1       Running   0          23h
esmaster-1-4205943399-zj2pn               1/1       Running   0          23h
esmaster-2-4248445829-pwq46               1/1       Running   0          23h
esmaster-3-3967126695-w0tp2               1/1       Running   0          23h
stackagent-3122989159-15vj1               1/1       Running   0          18h

Answer 1

插件configuration.The ">/stats/">的API问题将只返回_local节点的信息，该节点是一个没有文档的客户端。

此外，Stackdriver数据将在k8s集群节点下收集，而不是在pod名称下进行收集。

部分解决方案是在数据节点中设置sidecar，并用相应的ES节点名称对elasticsearch.conf查询进行修补：

1. 获取curl [elasticsearch]:9200/_nodes/stats
2. 通过$(主机名)查找ES节点名
3. 修补配置<URL "http://elasticsearch:9200/_nodes/<esnode_name>/stats/">

这将在k8s数据节点名称下收集ES数据节点的信息。