livenessprobe失败，EOF (nginx容器)

Question

我有一个运行nginx的容器，它监听pod id的端口443。它本身运行得很好；但是，如果我指定了一个活动探测器，那么探测器将失败，并显示

5m54s       Warning   Unhealthy          Pod           Liveness probe failed: Get https://192.168.2.243:443/: EOF

有人能指出我做错了什么吗？谢谢。

当它在没有活动探测的情况下运行时：

root@ip-192-168-2-243:/etc/nginx# netstat -tupln | grep 443
tcp        0      0 192.168.2.243:1443      0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.2.243:443       0.0.0.0:*               LISTEN      7/nginx: master pro

root@ip-192-168-2-243:/# telnet 192.168.2.243 443
Trying 192.168.2.243...
Connected to 192.168.2.243.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

root@ip-192-168-2-243:/# curl https://192.168.2.243
curl: (77) error setting certificate verify locations:
  CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs

探测声明：

livenessProbe:
  initialDelaySeconds: 10
  timeoutSeconds: 4
  failureThreshold: 3
  httpGet:
    scheme: HTTPS
    port: 443

Nginx拆分客户端声明：

split_clients "${remote_addr}AAA" $localips {
       *                 192.168.2.243;
}

数据平面/kubelet.service-ip：

​

事件：

skwok-mbp:kubernetes skwok$ kubectl get event -w
LAST SEEN   TYPE     REASON             OBJECT              MESSAGE
7s          Normal   SuccessfulDelete   statefulset/mnsvr   delete Pod mnsvr-0 in StatefulSet mnsvr successful
0s          Normal   Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-proxy:Need to kill Pod
0s          Normal   Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-node0:Need to kill Pod
0s          Normal   Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-node1:Need to kill Pod
0s          Normal   SuccessfulCreate   statefulset/mnsvr   create Pod mnsvr-0 in StatefulSet mnsvr successful
0s          Normal   Scheduled          pod/mnsvr-0         Successfully assigned staging/mnsvr-0 to ip-192-168-2-243.us-west-2.compute.internal
0s          Normal   Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr-proxy:0.96" already present on machine
0s          Normal   Created            pod/mnsvr-0         Created container
0s          Normal   Started            pod/mnsvr-0         Started container
0s          Normal   Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr:1.1" already present on machine
0s          Normal   Created            pod/mnsvr-0         Created container
0s          Normal   Started            pod/mnsvr-0         Started container
0s          Normal   Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr:1.1" already present on machine
0s          Normal   Created            pod/mnsvr-0         Created container
0s          Normal   Started            pod/mnsvr-0         Started container
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Normal    Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-proxy:Container failed liveness probe.. Container will be killed and recreated.
0s          Normal    Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr-proxy:0.96" already present on machine
0s          Normal    Created            pod/mnsvr-0         Created container
0s          Normal    Started            pod/mnsvr-0         Started container
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Normal    Killing            pod/mnsvr-0         Killing container with id docker://mnsvr-proxy:Container failed liveness probe.. Container will be killed and recreated.
0s          Normal    Pulled             pod/mnsvr-0         Container image "171421899218.dkr.ecr.us-west-2.amazonaws.com/mnsvr-proxy:0.96" already present on machine
0s          Normal    Created            pod/mnsvr-0         Created container
0s          Normal    Started            pod/mnsvr-0         Started container
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   Unhealthy          pod/mnsvr-0         Liveness probe failed: Get https://192.168.2.243:443/: EOF
0s          Warning   BackOff            pod/mnsvr-0         Back-off restarting failed container

host/host.messages-ip-192-168-2-243：

​

application/mnsvr-proxy：

​

Answer 1

我认为EOF是TLS握手问题的症状。我现在也看到了同样的情况。

curl的某些版本可以产生类似的结果。curl的一种解决方法似乎是使用--tls-max1.2。

我目前的怀疑是客户端(探测器)试图与服务器协商TLS 1.3，但失败了(可能是由于密码)。我正在尝试看看是否可以将k8s探测器配置为使用TLS1.2。或者，我们可以在服务器端关闭TLS 1.3。在你的情况下是在nginx上。在我的例子中，我有一个带有JDK 11.0.6的jetty 9.4服务器。

另一种选择可能是升级k8s。我们似乎在k8s v1.15集群中看到了这一点，但在k8s v1.16.2集群中却没有。但我不确定这是由于k8s版本还是底层的OS库(在我的例子中是CentOS 7)。

Answer 2

Kubernetes有两种不同的方法来跟踪pod的健康状况，一种是在部署期间，另一种是在部署之后。LivenessProbe是导致Kubernetes用新的pod替换失败的pod的原因，但它在应用程序部署期间绝对没有影响。另一方面，就绪探测是Kubernetes用来确定pod是否成功启动的工具。

因此，在您的容器成功工作的情况下，您必须定义readinessProbe。

有时，应用程序暂时无法为流量提供服务。例如，应用程序可能需要在启动期间加载大型数据或配置文件，或者在启动后依赖于外部服务。在这种情况下，您不想终止应用程序，但也不想向它发送请求。Kubernetes提供了就绪探测来检测和缓解这些情况。具有容器报告其未准备就绪的pod无法通过Kubernetes服务接收流量。

描述探测器的官方kubernetes文档：kubernetes-probes。

这里有一篇有用的文章：kubernetes-liveness-and-readiness-probes。