由于已经使用的NodePort，Kubernetes一直在删除Heapster & Grafana服务。

Question

我正在通过Docker在Ubuntu (信任的)本地运行一个Kubernetes集群。

由于我使用Vagrant来创建Ubuntu，所以我不得不从正式的Kubernetes指南中修改docker run命令：

docker run -d \
    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:rw \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --privileged=true \
    --pid=host \
    gcr.io/google_containers/hyperkube:v1.3.0 \
    /hyperkube kubelet \
        --allow-privileged=true \
        --api-servers=http://localhost:8080 \
        --v=2 \
        --address=0.0.0.0 \
        --enable-server \
        --hostname-override=192.168.10.30 \
        --config=/etc/kubernetes/manifests-multi \
        --containerized \
        --cluster-dns=10.0.0.10 \
        --cluster-domain=cluster.local

此外，运行反向代理允许我通过VM外部的浏览器访问集群的服务：

docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.3.0 \
/hyperkube proxy --master=http://127.0.0.1:8080 --v=2

这些步骤工作得很好，最终我可以在浏览器中访问Kubernetes UI。

vagrant@trusty-vm:~$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

现在，我想在这个具有InfluxDB后端和Grafana UI的Kubernetes集群中运行Heapster，正如在本指南中所描述的那样。为此，我克隆了Heapster，并通过添加grafana-service.yaml将type: NodePort配置为使用外部IP

apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-grafana
  name: monitoring-grafana
  namespace: kube-system
spec:
  # In a production setup, we recommend accessing Grafana through an external Loadbalancer
  # or through a public IP. 
  type: NodePort
  ports:
  - port: 80
    targetPort: 3000
  selector:
    name: influxGrafana

创建服务、rcs等：

vagrant@trusty-vm:~/heapster$ kubectl create -f deploy/kube-config/influxdb/
You have exposed your service on an external port on all nodes in your
cluster.  If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:30593) to serve traffic.

See http://releases.k8s.io/release-1.3/docs/user-guide/services-firewalls.md for more details.
service "monitoring-grafana" created
replicationcontroller "heapster" created
service "heapster" created
replicationcontroller "influxdb-grafana" created
service "monitoring-influxdb" created

vagrant@trusty-vm:~/heapster$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
Heapster is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
monitoring-grafana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana

vagrant@trusty-vm:~/heapster$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                READY     STATUS              RESTARTS   AGE
kube-system   heapster-y2yci                      1/1       Running             0          32m
kube-system   influxdb-grafana-6udas              2/2       Running             0          32m
kube-system   k8s-master-192.168.10.30            4/4       Running             0          58m
kube-system   k8s-proxy-192.168.10.30             1/1       Running             0          58m
kube-system   kube-addon-manager-192.168.10.30    2/2       Running             0          57m
kube-system   kube-dns-v17-y4cwh                  3/3       Running             0          58m
kube-system   kubernetes-dashboard-v1.1.0-bnbnp   1/1       Running             0          58m

vagrant@trusty-vm:~/heapster$ kubectl get svc --all-namespaces
NAMESPACE     NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
default       kubernetes             10.0.0.1     <none>        443/TCP             18m
kube-system   heapster               10.0.0.234   <none>        80/TCP              3s
kube-system   kube-dns               10.0.0.10    <none>        53/UDP,53/TCP       18m
kube-system   kubernetes-dashboard   10.0.0.58    <none>        80/TCP              18m
kube-system   monitoring-grafana     10.0.0.132   <nodes>       80/TCP              3s
kube-system   monitoring-influxdb    10.0.0.197   <none>        8083/TCP,8086/TCP   16m

如您所见，一切看起来都很顺利，我还可以通过浏览器访问Grafana在http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/上的UI。

然而，在大约1分钟后，Heapster和Grafana端点从cluster-info. kubectl消失。

vagrant@trusty-vm:~/heapster$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

浏览器输出：

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "endpoints \"monitoring-grafana\" not found",
  "reason": "NotFound",
  "details": {
    "name": "monitoring-grafana",
    "kind": "endpoints"
  },
  "code": 404
}

吊舱还在运行..。

vagrant@trusty-vm:~/heapster$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                READY     STATUS              RESTARTS   AGE
kube-system   heapster-y2yci                      1/1       Running             0          32m
kube-system   influxdb-grafana-6udas              2/2       Running             0          32m
kube-system   k8s-master-192.168.10.30            4/4       Running             0          58m
kube-system   k8s-proxy-192.168.10.30             1/1       Running             0          58m
kube-system   kube-addon-manager-192.168.10.30    2/2       Running             0          57m
kube-system   kube-dns-v17-y4cwh                  3/3       Running             0          58m
kube-system   kubernetes-dashboard-v1.1.0-bnbnp   1/1       Running             0          58m

..。但是Heapster和Grafana服务已经消失了：

vagrant@trusty-vm:~/heapster$ kubectl get svc --all-namespaces
NAMESPACE     NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
default       kubernetes             10.0.0.1     <none>        443/TCP             19m
kube-system   kube-dns               10.0.0.10    <none>        53/UDP,53/TCP       19m
kube-system   kubernetes-dashboard   10.0.0.58    <none>        80/TCP              19m
kube-system   monitoring-influxdb    10.0.0.197   <none>        8083/TCP,8086/TCP   17m

在检查kubectl cluster-info dump 的输出时，我发现了以下错误：

I0713 09:31:09.088567       1 proxier.go:427] Adding new service "kube-system/monitoring-grafana:" at 10.0.0.227:80/TCP
E0713 09:31:09.273385       1 proxier.go:887] can't open "nodePort for kube-system/monitoring-grafana:" (:30593/tcp), skipping this nodePort: listen tcp :30593: bind: address alread$
I0713 09:31:09.395280       1 proxier.go:427] Adding new service "kube-system/heapster:" at 10.0.0.111:80/TCP
E0713 09:31:09.466306       1 proxier.go:887] can't open "nodePort for kube-system/monitoring-grafana:" (:30593/tcp), skipping this nodePort: listen tcp :30593: bind: address alread$
I0713 09:31:09.480468       1 proxier.go:502] Setting endpoints for "kube-system/monitoring-grafana:" to [172.17.0.5:3000]
E0713 09:31:09.519698       1 proxier.go:887] can't open "nodePort for kube-system/monitoring-grafana:" (:30593/tcp), skipping this nodePort: listen tcp :30593: bind: address alread$
I0713 09:31:09.532026       1 proxier.go:502] Setting endpoints for "kube-system/heapster:" to [172.17.0.4:8082]
E0713 09:31:09.558527       1 proxier.go:887] can't open "nodePort for kube-system/monitoring-grafana:" (:30593/tcp), skipping this nodePort: listen tcp :30593: bind: address alread$
E0713 09:31:17.249001       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:31:22.252280       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:31:27.257895       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:31:31.126035       1 proxier.go:887] can't open "nodePort for kube-system/monitoring-grafana:" (:30593/tcp), skipping this nodePort: listen tcp :30593: bind: address alread$
E0713 09:31:32.264430       1 server.go:294] Starting health server failed: E0709 09:32:01.153168       1 proxier.go:887] can't open "nodePort for kube-system/monitoring-grafana:" ($
E0713 09:31:37.265109       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:31:42.269035       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:31:47.270950       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:31:52.272354       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:31:57.273424       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
E0713 09:32:01.153168       1 proxier.go:887] can't open "nodePort for kube-system/monitoring-grafana:" (:30593/tcp), skipping this nodePort: listen tcp :30593: bind: address alread$
E0713 09:32:02.276318       1 server.go:294] Starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use
I0713 09:32:06.105878       1 proxier.go:447] Removing service "kube-system/monitoring-grafana:"
I0713 09:32:07.175025       1 proxier.go:447] Removing service "kube-system/heapster:"
I0713 09:32:07.210270       1 proxier.go:517] Removing endpoints for "kube-system/monitoring-grafana:"
I0713 09:32:07.249824       1 proxier.go:517] Removing endpoints for "kube-system/heapster:"

显然，由于nodePort已经在使用，Heapster & Grafana的服务和端点被删除了。我没有在grafana-service.yaml中指定一个指定的grafana-service.yaml，这意味着Kubernetes可以选择一个还没有使用的--那么这怎么会是一个错误呢？还有，有什么办法解决这个问题吗？

操作系统:Ubuntu14.04.4 LTS (可信)\x{e76f} Kubernetes: v1.3.0 LTS: v1.11.2

Answer 1

我遇到了一个非常类似的问题。

在grafana-service.yaml文件(可能还有heapster-service.yaml文件)中，您有一行：kubernetes.io/cluster-service: 'true'。此标签意味着此服务将由addon管理器管理。当addon-manager运行它的定期检查时，它将看到没有在/etc/kubernetes/addons中定义的grafana/heapster服务，并将删除该服务。

要解决这个问题，您有两个选择：

1. 将标签更改为kubernetes.io/cluster-service: 'false'。
2. 将控制器和服务yaml文件移动到主节点上的/etc/kubernetes/addons (或任何配置addon管理器以查找yaml文件的位置)。

希望这有帮助

Answer 2

在我们的环境中同样的问题。K8S版本= 1.3.4，Docker1.12，Heapster是主分支的最新版本