k3s:驱逐管理器:试图回收“resourceName=”临时存储

Question

当我运行kubectl get pods -w --namespace kubeapps时，我看到我所有的吊舱都挂起了

NAME                                                         READY   STATUS    RESTARTS   AGE
kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
kubeapps-postgresql-0                                        0/1     Pending   0          19h
kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h

当我检查journalctl -u k3s时，我看到

E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object "kube-system"/"chart-content>
E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object "kube-system"/"chart-con>
E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object "kube-system"/"chart-values-t>
E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object "kube-system"/"chart-valu>
E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
I0604 11:37:08.079010    1301 eviction_manager.go:338] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.079027    1301 container_gc.go:85] "Attempting to delete unused containers"
I0604 11:37:08.079496    1301 image_gc_manager.go:327] "Attempting to delete unused images"
I0604 11:37:08.082310    1301 eviction_manager.go:349] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.082338    1301 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[kube-system/coredns-d76bd69b-2svkn >
E0604 11:37:08.082353    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/coredns-d76bd69b-2svk>
E0604 11:37:08.082359    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/metrics-server-7cd5fc>
E0604 11:37:08.082363    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/local-path-provisione>
I0604 11:37:08.082368    1301 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node"

我认为这是驱逐管理器的结果，因为我的系统磁盘空间不足？也许它只是免费的，下面是来自df -h的设备

我在错误日志中也看到了这个，

k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes"

Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /

k3s上的这些设置在哪里？

Answer 1

这是因为驱逐管理器的默认设置(当前硬设置为5%，图像回收最小设置为10%)保护了一个新的吊舱，耗尽了硬盘上的空间，而且度量标准有点过于保守。您可以通过提供一个覆盖--kubelet-arg的k3s缺省值。来更改驱逐管理器设置。可以通过提供kubelet-arg给你的k3sKUBECONFIG在…/etc/rancher/k3s/config.yaml来持久化。

像这样，

kubelet-arg:
  - "eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%"
  - "eviction-hard=memory.available<500Mi,nodefs.available<10Gi"

注意，尽管当前需要，但在上游Kubernetes中，通过命令行将选项传递给k3s是不可取的，并且当/如果它最终被删除时，它将在k3s中被删除.