kube-scheduler活动探测失败: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect:连接被拒绝

Question

所以我让这个不健康的集群部分地在数据中心工作。这可能是我第10次根据https://kubernetes.io/docs/setup/independent/high-availability/上的说明进行重建。

我可以将一些pod应用到这个集群，它似乎可以工作，但最终它开始变慢并崩溃，正如你在下面看到的。以下是调度程序清单：

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    image: k8s.gcr.io/kube-scheduler:v1.14.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

$ kubectl -n kube-系统获取pods

NAME                                       READY   STATUS             RESTARTS   AGE
coredns-fb8b8dccf-42psn                    1/1     Running            9          88m
coredns-fb8b8dccf-x9mlt                    1/1     Running            11         88m
docker-registry-dqvzb                      1/1     Running            1          2d6h
kube-apiserver-kube-apiserver-1            1/1     Running            44         2d8h
kube-apiserver-kube-apiserver-2            1/1     Running            34         2d7h
kube-controller-manager-kube-apiserver-1   1/1     Running            198        2d2h
kube-controller-manager-kube-apiserver-2   0/1     CrashLoopBackOff   170        2d7h
kube-flannel-ds-amd64-4mbfk                1/1     Running            1          2d7h
kube-flannel-ds-amd64-55hc7                1/1     Running            1          2d8h
kube-flannel-ds-amd64-fvwmf                1/1     Running            1          2d7h
kube-flannel-ds-amd64-ht5wm                1/1     Running            3          2d7h
kube-flannel-ds-amd64-rjt9l                1/1     Running            4          2d8h
kube-flannel-ds-amd64-wpmkj                1/1     Running            1          2d7h
kube-proxy-2n64d                           1/1     Running            3          2d7h
kube-proxy-2pq2g                           1/1     Running            1          2d7h
kube-proxy-5fbms                           1/1     Running            2          2d8h
kube-proxy-g8gmn                           1/1     Running            1          2d7h
kube-proxy-wrdrj                           1/1     Running            1          2d8h
kube-proxy-wz6gv                           1/1     Running            1          2d7h
kube-scheduler-kube-apiserver-1            0/1     CrashLoopBackOff   198        2d2h
kube-scheduler-kube-apiserver-2            1/1     Running            5          18m
nginx-ingress-controller-dz8fm             1/1     Running            3          2d4h
nginx-ingress-controller-sdsgg             1/1     Running            3          2d4h
nginx-ingress-controller-sfrgb             1/1     Running            1          2d4h

$ kubectl apiserver- -n描述pod kube--n-kube-apiserver-1

Containers:
  kube-scheduler:
    Container ID:  docker://c04f3c9061cafef8749b2018cd66e6865d102f67c4d13bdd250d0b4656d5f220
    Image:         k8s.gcr.io/kube-scheduler:v1.14.2
    Image ID:      docker-pullable://k8s.gcr.io/kube-scheduler@sha256:052e0322b8a2b22819ab0385089f202555c4099493d1bd33205a34753494d2c2
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-scheduler
      --bind-address=127.0.0.1
      --kubeconfig=/etc/kubernetes/scheduler.conf
      --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
      --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
      --leader-elect=true
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 28 May 2019 23:16:50 -0400
      Finished:     Tue, 28 May 2019 23:19:56 -0400
    Ready:          False
    Restart Count:  195
    Requests:
      cpu:        100m
    Liveness:     http-get http://127.0.0.1:10251/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/kubernetes/scheduler.conf from kubeconfig (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kubeconfig:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/scheduler.conf
    HostPathType:  FileOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:
  Type     Reason          Age                    From                       Message
  ----     ------          ----                   ----                       -------
  Normal   Created         4h56m (x104 over 37h)  kubelet, kube-apiserver-1  Created container kube-scheduler
  Normal   Started         4h56m (x104 over 37h)  kubelet, kube-apiserver-1  Started container kube-scheduler
  Warning  Unhealthy       137m (x71 over 34h)    kubelet, kube-apiserver-1  Liveness probe failed: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
  Normal   Pulled          132m (x129 over 37h)   kubelet, kube-apiserver-1  Container image "k8s.gcr.io/kube-scheduler:v1.14.2" already present on machine
  Warning  BackOff         128m (x1129 over 34h)  kubelet, kube-apiserver-1  Back-off restarting failed container
  Normal   SandboxChanged  80m                    kubelet, kube-apiserver-1  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          76m                    kubelet, kube-apiserver-1  Error: context deadline exceeded
  Normal   Pulled          36m (x7 over 78m)      kubelet, kube-apiserver-1  Container image "k8s.gcr.io/kube-scheduler:v1.14.2" already present on machine
  Normal   Started         36m (x6 over 74m)      kubelet, kube-apiserver-1  Started container kube-scheduler
  Normal   Created         32m (x7 over 74m)      kubelet, kube-apiserver-1  Created container kube-scheduler
  Warning  Unhealthy       20m (x9 over 40m)      kubelet, kube-apiserver-1  Liveness probe failed: Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
  Warning  BackOff         2m56s (x85 over 69m)   kubelet, kube-apiserver-1  Back-off restarting failed container

我觉得我忽略了一个简单的选项或配置，但我找不到它，在处理这个问题和阅读文档几天后，我束手无策。

负载均衡器是一个TCP负载均衡器，似乎工作正常，因为我可以从我的桌面查询集群。

现在绝对欢迎任何建议或故障排除技巧。

谢谢。

Answer 1

我们配置的问题是，一位意图良好的技术人员决定取消kubernetes主防火墙上的一条规则，该规则阻止了主防火墙环回到它需要探测的端口。这导致了各种奇怪的问题和误诊问题，这肯定是错误的方向。在我们允许服务器上的所有端口之后，Kubernetes恢复了它的正常行为。