在更新主机中的GPU驱动程序后，旧的码头容器无法使用(没有GPU)

Question

今天，我们更新了主机的GPU驱动程序，我们创建的新容器都正常工作。但是，所有现有的坞容器在内部运行nvidia-smi命令时都会出现以下错误：

未能初始化NVML:驱动程序/库版本不匹配

如何拯救这些旧容器？我们以前在主机上的GPU驱动程序版本是384.125，现在是430.64。

主机配置

nvidia-smi给出

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-DGXS...  Off  | 00000000:07:00.0  On |                    0 |
| N/A   40C    P0    39W / 300W |    182MiB / 32505MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-DGXS...  Off  | 00000000:08:00.0 Off |                    0 |
| N/A   40C    P0    39W / 300W |     12MiB / 32508MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-DGXS...  Off  | 00000000:0E:00.0 Off |                    0 |
| N/A   39C    P0    40W / 300W |     12MiB / 32508MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-DGXS...  Off  | 00000000:0F:00.0 Off |                    0 |
| N/A   40C    P0    38W / 300W |     12MiB / 32508MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1583      G   /usr/lib/xorg/Xorg                           169MiB |
+-----------------------------------------------------------------------------+

nvcc --version给出

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

dpkg -l | grep -i docker给出

ii  dgx-docker-cleanup                         1.0-1                                           amd64        DGX Docker cleanup script
rc  dgx-docker-options                         1.0-7                                           amd64        DGX docker daemon options
ii  dgx-docker-repo                            1.0-1                                           amd64        docker repository configuration file
ii  docker-ce                                  5:18.09.2~3-0~ubuntu-xenial                     amd64        Docker: the open-source application container engine
ii  docker-ce-cli                              5:18.09.2~3-0~ubuntu-xenial                     amd64        Docker CLI: the open-source application container engine
ii  nvidia-container-runtime                   2.0.0+docker18.09.2-1                           amd64        NVIDIA container runtime
ii  nvidia-docker                              1.0.1-1                                         amd64        NVIDIA Docker container tools
rc  nvidia-docker2                             2.0.3+docker18.09.2-1                           all          nvidia-docker CLI wrapper

docker version给出

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        6247962
 Built:             Sun Feb 10 04:13:50 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:42:13 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Answer 1

我也遇到了这个问题。就我而言，我的底线是：

apt install -y nvidia-cuda-toolkit

在我的Dockerfile里。删除这一行解决了这个问题。通常，我建议使用与本地计算机上的驱动程序兼容的nvidia提供的集装箱。