How to Configure NVIDIA GPUs with Docker on Ubuntu: A Comprehensive Guide for AI Deep Learning CUDA Solution

Welcome to our in-depth guide on configuring NVIDIA GPUs with Docker on Ubuntu. This post is tailored for developers, data scientists, and IT professionals who are looking to leverage the power of NVIDIA's GPU acceleration within Docker containers. 

Whether you're working on machine learning projects, scientific computations, or any GPU-intensive tasks, this guide will walk you through the process step-by-step.

This guide will work for all Nvidia GPUs that have a supported driver in Linux such as the GTX, RTX and Tesla series.  Of course the Tesla series is recommended as they have ECC and are more tailored for AI applications.

Normally it is not possible to share a GPU with multiple containers

Typically 1 real/physical GPU can work on one machine, whether physical or a VM that gets exclusive use.  But nvidia has made tools that solve this problem by essentially creating a layer between Docker and the nvidia driver.

There is the old nvidia-docker2 package that NVIDIA has created which allows an unlimited amount of Docker containers to use the underlaying GPU(s) on the host  but this has now been deprecrated for the new "nvidia-toolkit"

nvidia-toolkit is what you want unless there's a reason why you have an older distro that can't use the newer nvidia-toolkit.

nvidia-toolkit official install guide:

How To Install nvidia-toolkit on Mint Ubuntu Debian

Step 1.) Add the Nvidia repo to Linux

First we add the gpg key for the repo:

curl -fsSL | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

Add the repo into our sources.list.d

curl -s -L | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |  
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update apt so it can see the required packages from the nvidia repo

apt update

Step 2 - Install nvidia toolkit

apt-get install -y nvidia-container-toolkit

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libnvidia-container-tools libnvidia-container1 nvidia-container-toolkit-base
The following NEW packages will be installed:
  libnvidia-container-tools libnvidia-container1 nvidia-container-toolkit nvidia-container-toolkit-base
0 upgraded, 4 newly installed, 0 to remove and 690 not upgraded.
Need to get 4,194 kB of archives.
After this operation, 16.6 MB of additional disk space will be used.
Get:1  libnvidia-container1 1.14.3-1 [923 kB]
Get:2  libnvidia-container-tools 1.14.3-1 [19.3 kB]
Get:3  nvidia-container-toolkit-base 1.14.3-1 [2,336 kB]
Get:4  nvidia-container-toolkit 1.14.3-1 [917 kB]
Fetched 4,194 kB in 1s (3,150 kB/s)           
Selecting previously unselected package libnvidia-container1:amd64.
(Reading database ... 467373 files and directories currently installed.)
Preparing to unpack .../libnvidia-container1_1.14.3-1_amd64.deb ...
Unpacking libnvidia-container1:amd64 (1.14.3-1) ...
Selecting previously unselected package libnvidia-container-tools.
Preparing to unpack .../libnvidia-container-tools_1.14.3-1_amd64.deb ...
Unpacking libnvidia-container-tools (1.14.3-1) ...
Selecting previously unselected package nvidia-container-toolkit-base.
Preparing to unpack .../nvidia-container-toolkit-base_1.14.3-1_amd64.deb ...
Unpacking nvidia-container-toolkit-base (1.14.3-1) ...
Selecting previously unselected package nvidia-container-toolkit.
Preparing to unpack .../nvidia-container-toolkit_1.14.3-1_amd64.deb ...
Unpacking nvidia-container-toolkit (1.14.3-1) ...
Setting up nvidia-container-toolkit-base (1.14.3-1) ...
Setting up libnvidia-container1:amd64 (1.14.3-1) ...
Setting up libnvidia-container-tools (1.14.3-1) ...
Setting up nvidia-container-toolkit (1.14.3-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...

Step 3 - Docker Configuration

nvidia-ctk runtime configure --runtime=docker

INFO[0000] Loading config from /etc/docker/daemon.json  
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that docker daemon be restarted.

Step 4 - Test

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

| NVIDIA-SMI 390.154                Driver Version: 390.154                   |
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|   0  Tesla V100-SXM2...      Off  | 00000000:04:00.0 N/A |              N/A |
| N/A   41C    P0    N/A /  N/A |    0MiB /   16160MiB |     N/A      Default |
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|    0                    Not Supported                                       |



configure, nvidia, gpus, docker, ubuntu, comprehensive, ai, cuda, solutionwelcome, depth, configuring, tailored, developers, scientists, professionals, leverage, gpu, acceleration, containers, projects, scientific, computations, intensive, tasks, supported, linux, gtx, rtx, tesla, recommended, ecc, applications, multiple, typically, vm, exclusive, essentially, creating, layer, allows, unlimited, underlaying, deprecrated, quot, toolkit, distro, newer, install, https, docs, datacenter, native, container, html, mint, debian, repo, gpg, curl, fssl, github, io, libnvidia, gpgkey, sudo, dearmor, usr, keyrings, keyring, sources, deb, sed, tee, etc, apt, update, packages, lists, dependency, additional, installed, upgraded, newly, kb, archives, mb, disk, amd, fetched, selecting, previously, unselected, database, directories, currently, preparing, unpack, _, _amd, unpacking, tools_, base_, toolkit_, processing, triggers, libc, bin, configuration, ctk, runtime, info, loading, config, daemon, json, updated, restarted, rm, smi, persistence, disp, volatile, uncorr, temp, perf, pwr, usage, util, compute, sxm, mib, default, processes, pid,

Latest Articles

  • How to resize a pdf without losing much quality in Linux Mint Ubuntu Debian Redhat Solution
  • qemu: could not load PC BIOS 'bios-256k.bin' solution
  • Proxmox How To Custom Partition During Install
  • Hyper-V Linux VM Boots to Black Screen, Storage, NIC Not Found Issues
  • Ubuntu Mint How to Fix Missing/Broken /dev and /dev/pts which causes terminal to immediately close exit and not work
  • How high can a Xeon CPU get?
  • bash fix PATH environment variable "command not found" solution
  • Ubuntu Linux Mint Debian Redhat Youtube Cannot Play HD or 4K videos, dropped frames or high CPU usage with Nvidia or AMD Driver
  • hostapd example configuration for high speed AC on 5GHz using WPA2
  • hostapd how to enable and use WPS to connect wireless devices like printers
  • Dell Server Workstation iDRAC Dead after Firmware Update Solution R720, R320, R730
  • Cloned VM/Server/Computer in Linux won't boot and goes to initramfs busybox Solution
  • How To Add Windows 7 8 10 11 to GRUB Boot List Dual Booting
  • How to configure OpenDKIM on Linux with Postfix and setup bind zonefile
  • Debian Ubuntu 10/11/12 Linux how to get tftpd-hpa server setup tutorial
  • efibootmgr: option requires an argument -- 'd' efibootmgr version 15 grub-install.real: error: efibootmgr failed to register the boot entry: Operation not permitted.
  • Apache Error Won't start SSL Cert Issue Solution Unable to configure verify locations for client authentication SSL Library Error: 151441510 error:0906D066:PEM routines:PEM_read_bio:bad end line SSL Library Error: 185090057 error:0B084009:x509 certif
  • Linux Debian Mint Ubuntu Bridge br0 gets random IP
  • redis requirements
  • How to kill a docker swarm