Enabling NVIDIA GPU Support in Docker with nvidia-container-toolkit

* This page contains promotional content

TL;DR

To answer the request to use a GPU (nvidia) inside a Docker container, Docker has to be configured for GPU support.

Normally a Docker container cannot access the host’s GPU resources. That is because Docker does not pass the host’s GPU devices through to the container, and the reason behind it is that the GPU driver and the CUDA libraries are not properly shared between the host and the container.
To get around this, the GPU resources have to be passed through to the container with a tool such as nvidia-container-toolkit.

Checking the GPU

Before making Docker GPU-capable, check the GPU in the PC.

$ lspci -k | grep -i nvidia
65:00.0 VGA compatible controller: NVIDIA Corporation GA104GL [RTX A4000] (rev a1)
	Subsystem: NVIDIA Corporation Device 14ad
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia
65:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
	Subsystem: NVIDIA Corporation Device 14ad

Installing the nvidia toolkit

Debian

$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

$ sudo apt update
$ sudo apt-get install -y nvidia-container-toolkit

$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker

CentOS7

$ sudo yum install -y yum-utils
$ sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/centos7/nvidia-docker.repo
$ sudo yum install -y nvidia-container-toolkit
$ sudo systemctl restart docker

AlmaLinux8

$ sudo dnf config-manager --add-repo https://nvidia.github.io/nvidia-docker/almalinux8/nvidia-docker.repo
$ sudo dnf clean expire-cache
$ sudo dnf install -y nvidia-container-toolkit
$ sudo systemctl restart docker

How to use a GPU with Docker

Docker command

To run a Docker container that uses a GPU, use the --gpus flag.

$ docker run --gpus all <image name>

The --gpus flag has the following options.

all: assigns every available GPU.
"device=<device ID>": assigns specific GPU devices (for example, “device=1,2”).
"<GPU-UUID>": assigns based on the UUID of the GPU.

docker-compose.yml

To use it from within docker-compose.yml, write the deploy section under the service as shown below.

version: "3.8"
services:
  your-service:
    image: <image name>
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

One GPU device is assigned to the service container.
By increasing count you can change the number of GPUs assigned.
Note that in order to use a GPU, the container image itself has to support GPUs.

docker

GPU nvidia