kubernetes: invalid_token授权失败，拉取镜像失败

Question

当我尝试拉取图像时，在kubernetes上收到以下错误消息：

0s          Normal    Pulling             pod/zeppelin-server-6f5646f56d-6slj2    Pulling image "apache/zeppelin:0.9.0"
0s          Warning   Failed              pod/zeppelin-server-6f5646f56d-6slj2    Failed to pull image "apache/zeppelin:0.9.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/apache/zeppelin:0.9.0": failed to copy: httpReaderSeeker: failed open: server message: invalid_token: authorization failed
0s          Warning   Failed              pod/zeppelin-server-6f5646f56d-6slj2    Error: ErrImagePull

这里是kubectl get events -w

<unknown>   Normal    Scheduled           pod/zeppelin                           Successfully assigned zeppelin/zeppelin to k3d-k3s-linkerd-server
47s         Normal    Pulling             pod/zeppelin                           Pulling image "apache/zeppelin:0.9.0"
19s         Warning   Failed              pod/zeppelin-server-6f5646f56d-6slj2   Failed to pull image "apache/zeppelin:0.9.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/apache/zeppelin:0.9.0": failed to copy: httpReaderSeeker: failed open: server message: invalid_token: authorization failed
19s         Warning   Failed              pod/zeppelin-server-6f5646f56d-6slj2   Error: ErrImagePull

另一边：

$ kubectl run zeppelin --image=apache/zeppelin:0.9.0 --restart=Never -v=10
I0531 12:13:29.882735   18907 loader.go:375] Config loaded from file:  /home/jeusdi/.config/k3d/k3s-linkerd/kubeconfig.yaml
I0531 12:13:29.884168   18907 cached_discovery.go:114] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/servergroups.json
I0531 12:13:29.884623   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/autoscaling/v1/serverresources.json
I0531 12:13:29.884623   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/autoscaling/v2beta1/serverresources.json
I0531 12:13:29.884633   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/authentication.k8s.io/v1/serverresources.json
I0531 12:13:29.884693   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/storage.k8s.io/v1/serverresources.json
I0531 12:13:29.884716   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/events.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.884728   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/linkerd.io/v1alpha2/serverresources.json
I0531 12:13:29.884773   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/authorization.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.884789   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/scheduling.k8s.io/v1/serverresources.json
I0531 12:13:29.884799   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/k3s.cattle.io/v1/serverresources.json
I0531 12:13:29.884826   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/v1/serverresources.json
I0531 12:13:29.884858   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/admissionregistration.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.884863   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/apps/v1/serverresources.json
I0531 12:13:29.884868   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/apiextensions.k8s.io/v1/serverresources.json
I0531 12:13:29.884894   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/caching.internal.knative.dev/v1alpha1/serverresources.json
I0531 12:13:29.884922   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/apiextensions.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.884929   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/batch/v1/serverresources.json
I0531 12:13:29.884922   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/apiregistration.k8s.io/v1/serverresources.json
I0531 12:13:29.884958   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/storage.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.884979   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/networking.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.884988   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/policy/v1beta1/serverresources.json
I0531 12:13:29.885002   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/apiregistration.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885008   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/flagger.app/v1beta1/serverresources.json
I0531 12:13:29.885025   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/batch/v1beta1/serverresources.json
I0531 12:13:29.885037   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/admissionregistration.k8s.io/v1/serverresources.json
I0531 12:13:29.885043   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/rbac.authorization.k8s.io/v1/serverresources.json
I0531 12:13:29.885069   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/extensions/v1beta1/serverresources.json
I0531 12:13:29.885076   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/certificates.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885085   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/flagger.app/v1alpha3/serverresources.json
I0531 12:13:29.885109   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885121   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/networking.k8s.io/v1/serverresources.json
I0531 12:13:29.885152   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/coordination.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885153   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/flagger.app/v1alpha1/serverresources.json
I0531 12:13:29.885182   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/linkerd.io/v1alpha1/serverresources.json
I0531 12:13:29.885182   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/tap.linkerd.io/v1alpha1/serverresources.json
I0531 12:13:29.885222   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/autoscaling/v2beta2/serverresources.json
I0531 12:13:29.885224   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/scheduling.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885247   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/split.smi-spec.io/v1alpha1/serverresources.json
I0531 12:13:29.885255   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/node.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885273   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/authentication.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885297   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/coordination.k8s.io/v1/serverresources.json
I0531 12:13:29.885310   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/discovery.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885311   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/tekton.dev/v1alpha1/serverresources.json
I0531 12:13:29.885363   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/traefik.containo.us/v1alpha1/serverresources.json
I0531 12:13:29.885365   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/tekton.dev/v1beta1/serverresources.json
I0531 12:13:29.885381   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/metrics.k8s.io/v1beta1/serverresources.json
I0531 12:13:29.885478   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/authorization.k8s.io/v1/serverresources.json
I0531 12:13:29.885705   18907 cached_discovery.go:71] returning cached discovery info from /home/jeusdi/.kube/cache/discovery/localhost_6443/helm.cattle.io/v1/serverresources.json
I0531 12:13:29.889299   18907 request.go:1068] Request Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"zeppelin","creationTimestamp":null,"labels":{"run":"zeppelin"}},"spec":{"containers":[{"name":"zeppelin","image":"apache/zeppelin:0.9.0","resources":{}}],"restartPolicy":"Never","dnsPolicy":"ClusterFirst"},"status":{}}
I0531 12:13:29.889359   18907 round_trippers.go:423] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.18.3 (linux/amd64) kubernetes/2e7996e" -H "Accept: application/json, */*" -H "Authorization: Basic YWRtaW46NjA4OGZhYmEzNjc3OGFkMGJhNzJiMzhiNzJjZTE5NWQ=" 'https://localhost:6443/api/v1/namespaces/zeppelin/pods'
I0531 12:13:29.912123   18907 round_trippers.go:443] POST https://localhost:6443/api/v1/namespaces/zeppelin/pods 201 Created in 22 milliseconds
I0531 12:13:29.912223   18907 round_trippers.go:449] Response Headers:
I0531 12:13:29.912233   18907 round_trippers.go:452]     Content-Length: 1241
I0531 12:13:29.912244   18907 round_trippers.go:452]     Content-Type: application/json
I0531 12:13:29.912250   18907 round_trippers.go:452]     Date: Sun, 31 May 2020 10:13:29 GMT
I0531 12:13:29.912324   18907 request.go:1068] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"zeppelin","namespace":"zeppelin","selfLink":"/api/v1/namespaces/zeppelin/pods/zeppelin","uid":"a01e2b82-2c47-4424-b1e0-d6bedcfe2d50","resourceVersion":"365387","creationTimestamp":"2020-05-31T10:13:29Z","labels":{"run":"zeppelin"}},"spec":{"volumes":[{"name":"default-token-49tq6","secret":{"secretName":"default-token-49tq6","defaultMode":420}}],"containers":[{"name":"zeppelin","image":"apache/zeppelin:0.9.0","resources":{},"volumeMounts":[{"name":"default-token-49tq6","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Pending","qosClass":"BestEffort"}}
pod/zeppelin created

问题只出现在apache/zeppelin上。我已经部署了其他pod：

$ kubectl get pods
NAME             READY   STATUS              RESTARTS   AGE
spark-master-0   1/1     Running             1          41h
spark-worker-1   1/1     Running             1          41h
spark-worker-0   1/1     Running             1          41h
zeppelin         0/1     ContainerCreating   0          5m14s

Answer 1

您可能希望登录到部署中的容器注册表(docker.io)，或者使用您的注册表凭据创建自定义dockerfile，并将其存储在容器中，然后使用该映像进行部署。

Answer 2

在测试过程中，我收到了几乎类似的授权问题“未授权:用户名或密码不正确”

kubectl run zeppelin --image=apache/zeppelin:0.9.0 --restart=Never -v=10


97s         Normal    Scheduled          pod/zeppelin                                             Successfully assigned default/zeppelin to gke-standard-cluster-1-default-pool-2584cac1-h3l7
82s         Normal    Pulling            pod/zeppelin                                             Pulling image "apache/zeppelin:0.9.0"
81s         Warning   Failed             pod/zeppelin                                             Failed to pull image "apache/zeppelin:0.9.0": rpc error: code = Unknown desc = Error response fr
om daemon: Get https://registry-1.docker.io/v2/apache/zeppelin/manifests/0.9.0: unauthorized: incorrect username or password
81s         Warning   Failed             pod/zeppelin                                             Error: ErrImagePull
67s         Normal    BackOff            pod/zeppelin                                             Back-off pulling image "apache/zeppelin:0.9.0"
67s         Warning   Failed             pod/zeppelin                                             Error: ImagePullBackOff
40s         Normal    Scheduled          pod/zeppelin                                             Successfully assigned default/zeppelin to gke-standard-cluster-1-default-pool-2584cac1-h3l7
26s         Normal    Pulling            pod/zeppelin                                             Pulling image "apache/zeppelin:0.9.0"
25s         Warning   Failed             pod/zeppelin                                             Failed to pull image "apache/zeppelin:0.9.0": rpc error: code = Unknown desc = Error response fr
om daemon: Get https://registry-1.docker.io/v2/apache/zeppelin/manifests/0.9.0: unauthorized: incorrect username or password
25s         Warning   Failed             pod/zeppelin                                             Error: ErrImagePull
12s         Normal    BackOff            pod/zeppelin                                             Back-off pulling image "apache/zeppelin:0.9.0"
12s         Warning   Failed             pod/zeppelin                                             Error: ImagePullBackOff

但是，我可以手动使用它

docker pull apache/zeppelin:0.9.0
0.9.0: Pulling from apache/zeppelin
fe703b657a32: Pull complete
f9df1fafd224: Pull complete
a645a4b887f9: Pull complete
57db7fe0b522: Pull complete
4e0427e9cf76: Pull complete
78257ba855d1: Pull complete
aeeac4e74c73: Pull complete
ec837c1cd2ab: Pull complete
9132860032fc: Pull complete
27fec663a415: Pull complete
98921bfcfa52: Pull complete
0f903898df05: Pull complete
e0ab70d3cb38: Pull complete
235cf5a6a0b9: Pull complete
Digest: sha256:0691909f6884319d366f5d3a5add8802738d6240a83b2e53e980caeb6c658092
Status: Downloaded newer image for apache/zeppelin:0.9.0
docker.io/apache/zeppelin:0.9.0

使用--image-pull-policy=Never运行本地存储的已经获取的图像如何？

imagePullPolicy: Never:假设镜像存在于本地。不会尝试拉取图像。

这听起来像是一种变通办法，但现在找不到更好的解决方案。如果找到真正的解决方案，我将更新问题