内存使用量超过k8s资源限制

Question

redis实例部署在k8s集群中，内存限制为1G：

resources:
  limits:
    memory: 1Gi

并将redis maxmemory设置为1G，实际内存使用量约为550米：

info memory
# Memory
used_memory:577791152
used_memory_human:551.02M
used_memory_rss:611942400
used_memory_rss_human:583.59M
used_memory_peak:578877888
used_memory_peak_human:552.06M
used_memory_peak_perc:99.81%
used_memory_overhead:32108388
used_memory_startup:786608
used_memory_dataset:545682764
used_memory_dataset_perc:94.57%
total_system_memory:67555897344
total_system_memory_human:62.92G
used_memory_lua:41984
used_memory_lua_human:41.00K
maxmemory:1000000000
maxmemory_human:953.67M
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.06
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

我注意到，prometheus (公制sum by(container_name) (container_memory_working_set_bytes{namespace="$namespace",container_name!="",container_name=~"$container",container_name!="POD",pod_name="$pod"}))中的redis容器内存使用量超过了1G：

​

​

以下是问题所在：

1. 如何释放记忆？
2. container_memory_working_set_bytes和redis报告的used_memory_rss used_memory之间有什么关系

Answer 1

这是一个与k8s prometheus 问题有关的问题，prometheus实际上得到了一个加倍的结果：

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",cpu="total",endpoint="https-metrics",id="/kubepods/pod149c0943-bc55-4e5d-ac9d-59fedca56138/1e2f5a63ddba49cb13465383e0de66e50ce9f084b52975f9bea8d1d77b966b3c",image="sha256:8a74675d2c029efb8c574f6097f2494d0e6acb7427aeea874227be48d96a3fb2",instance="192.168.116.119:10250",job="expose-kubelets-metrics",name="k8s_redis-server_redis-server-0_user-data-service-test_149c0943-bc55-4e5d-ac9d-59fedca56138_0",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"}

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",endpoint="https-metrics",instance="192.168.116.119:10250",job="expose-kubelets-metrics",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"}    

所以结果加倍了。