内存使用模拟的“`nvidia smi”之间有什么区别？

Question

我得到nvidia-smi get Memory-Usage是这样的

$nvidia-smi -i 0,1
Wed Mar  4 16:20:07 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.113      Driver Version: 418.113      CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:18:00.0 Off |                  N/A |
| 27%   37C    P8     1W / 250W |  10789MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:3B:00.0 Off |                  N/A |
| 41%   50C    P2    67W / 250W |  10893MiB / 10989MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0    231853      C   tensorflow_model_server                    10779MiB |
|    1    120908      C   python                                     10883MiB |
+-----------------------------------------------------------------------------+

这个Memory-Usage是99%，但是当我得到这样的nvidia-smi dmon

$nvidia-smi dmon -i 0,1
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0     1    37     -     0     0     0     0   405   300
    1    67    50     -     0     0     0     0  6800  1350
    0     1    37     -     0     0     0     0   405   300
    1    67    50     -     0     0     0     0  6800  1350
    0     1    37     -     0     0     0     0   405   300
    1    67    50     -     0     0     0     0  6800  1350

该mem%为0%，有些时间为0~3%。

为何会有这样的分别呢？

Answer 1

Memory-Usage from nvidia-smi是内存的用法。

mem% from nvidia-smi dmon是内存的利用。

Memory-Usage = used memory / total memory.

Utilization = time over the past sample period / global (device) memory was being read or written * 100%