kube-apiserver拒绝端口443上的外部连接

Question

我有一个使用kube-aws配置的kubernetes集群。我添加了两个额外的控制器，并在三个控制器节点上集群了etcd。当我在不同的节点上设置和获取值时，etcd集群看起来(基本上)是健康的，并且工作正常。

我将新的etcd服务器添加到/etc/kubernetes/manifests/kube-apiserver.yaml：

--etcd-servers=http://10.0.0.50:2379,http://10.0.1.50:2379,http://10.0.2.50:2379

现在，kube-apiserver拒绝安全端口443上的连接。

kubelet日志包含这样的条目...

E0617 02:07:17.488289   30972 pod_workers.go:138] Error syncing pod 
b394008e8164, skipping: failed to "StartContainer" for "etcd-container" with 
CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd-container 
pod=etcd-server-ip-10-0-0-50.eu-west-1.compute.internal_default(b394008e8164)"

还有这个..。

E0617 02:03:00.542951   30972 event.go:193] Server rejected event 
'&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, 
ObjectMeta:api.ObjectMeta{Name:"etcd-server-ip-10-0-0-50.eu-
west-1.compute.internal.1458b", GenerateName:"", Namespace:"default", 
SelfLink:"", UID:"", ResourceVersion:"1189", Generation:0, 
CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, 
loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), 
DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), 
Annotations:map[string]string(nil)}, 
InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"default", 
Name:"etcd-server-ip-10-0-0-50.eu-west-1.compute.internal", 
UID:"b394008e8164", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, 
Reason:"FailedSync", Message:"Error syncing pod, skipping: failed to 
\"StartContainer\" for \"etcd-container\" with RunContainerError: 
\"runContainer: API error (500): mkdir /usr/local/openssl: read-only file 
system\\n\"\n", Source:api.EventSource{Component:"kubelet", Host:"ip-
10-0-0-50.eu-west-1.compute.internal"}, 
FirstTimestamp:unversioned.Time{Time:time.Time{sec:63601711997, nsec:0, 
loc:(*time.Location)(0x3b1a5c0)}}, 
LastTimestamp:unversioned.Time{Time:time.Time{sec:63601725780, 
nsec:522501285, loc:(*time.Location)(0x3b1a5c0)}}, Count:7, Type:"Warning"}': 
'events "etcd-server-ip-10-0-0-50.eu-west-1.compute.internal.1458b" not 
found' (will not retry!)

在端口8080本地查询kube-apiserver，如下所示...

$ curl http://localhost:8080/api/v1/namespaces

..。返回有效的响应。

但它拒绝在安全端口上进行任何尝试。

有谁能告诉我正确的方向吗？

Answer 1

对于任何发现这个问题与kube-apiserver有问题的人...

我的问题原来是驻留在/etc/kubernetes/ssl/中的不良TLS资产。(事后看来，这是显而易见的，因为它响应的是HTTP请求，而不是HTTPS。)我问题中的日志条目与TLS资产的问题无关。kubelet日志帮助我追踪到了这个问题：

$ journalctl -u kubelet -e  # jump to end of kubelet logs

以下内容也派上了用场：

$ journalctl -u oem-cloudinit  # if using aws will help you debug your cloud-config
$ journalctl -u etcd2 -f  # follow etcd logs