在etcd3中找不到Kubernetes apiserver的数据

Question

我正在使用来自kargo的ansible脚本来构建我的集群。我无法找到数据在etcd3中的存储位置，尽管查看了来自apiserver的详细日志。

下面是我看到的hyperkube apiserver日志：

$ docker logs k8s_kube-apiserver.fd19548d_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_aa25196e
[[ SNIP ]]
I0127 23:31:55.871267       1 storage_factory.go:242] storing { podtemplates} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.875975       1 storage_factory.go:242] storing { events} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.876169       1 reflector.go:234] Listing and watching *api.PodTemplate from k8s.io/kubernetes/pkg/storage/cacher.go:215
I0127 23:31:55.877950       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]
I0127 23:31:55.878148       1 storage_factory.go:242] storing { limitranges} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.879372       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]

hyperkube apiserver是通过以下参数启动的：

$ docker inspect k8s_kube-apiserver.b6395694_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_4338b366
[
    {
        "Id": "33c76fa64bbd5d5a656e329cf87ed3707077659c69dc281127751f594460242b",
        "Created": "2017-01-27T23:35:10.691147667Z",
        "Path": "/hyperkube",
        "Args": [
            "apiserver",
            "--advertise-address=10.60.68.23",
            "--etcd-servers=https://10.60.68.11:2379,https://10.60.68.39:2379,https://10.60.68.35:2379",
            "--etcd-quorum-read=true",
            "--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem",
            "--etcd-certfile=/etc/ssl/etcd/ssl/node-kube-master-01.pem",
            "--etcd-keyfile=/etc/ssl/etcd/ssl/node-kube-master-01-key.pem",
            "--insecure-bind-address=127.0.0.1",
            "--apiserver-count=3",
            "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota",
            "--service-cluster-ip-range=10.233.0.0/18",
            "--service-node-port-range=30000-32767",
            "--client-ca-file=/etc/kubernetes/ssl/ca.pem",
            "--basic-auth-file=/etc/kubernetes/users/known_users.csv",
            "--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem",
            "--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--token-auth-file=/etc/kubernetes/tokens/known_tokens.csv",
            "--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--secure-port=443",
            "--insecure-port=8080",
            "--v=4",
            "--allow-privileged=true",
            "--cloud-provider=openstack",
            "--cloud-config=/etc/kubernetes/cloud_config",
            "--anonymous-auth=False"
        ],

否覆盖/registry的默认etcd前缀。我不知道apiserver在哪里存储数据。

$ docker exec -it etcd3 etcdctl --peers https://10.60.68.11:2379 ls /registry
Error:  100: Key not found (/registry) [163]

Answer 1

要获取kubernetes存储在etcd v3中的键和值：

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get --prefix /registry

要从etcd v3中获取1个指定的键和值，例如：

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get /registry/services/specs/default/kubernetes

基于：https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md

Answer 2

如果您的系统设置正确，请注意从etcd2到etcd3有一些更改。例如，与etcd2不同，etcd3中的键空间现在是扁平的，因此不再有任何目录。不使用上面所示的etcdctl 'ls‘命令，而是使用'get’命令(并确保etcdctl命令通过设置ETCDCTL_API=3环境变量来使用v3 api )。

Answer 3

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get / --prefix --keys-only