Docker: could not select device driver with capabilities gpu — Every Fix for NVIDIA GPU Passthrough in 2026

By RunAIHome Team · 11 min read
dockernvidiagpuollamalocal-llmtroubleshooting

TL;DR: could not select device driver "" with capabilities: [[gpu]] almost always means the NVIDIA Container Toolkit isn’t installed or Docker wasn’t restarted after configuring it — not a broken driver. Install the toolkit, run nvidia-ctk runtime configure, restart Docker, and the GPU appears inside the container. On WSL2 the fix is the same, plus one trap that wastes an afternoon.

What you’ll be able to do after this:

  • Run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu24.04 nvidia-smi and see your card
  • Start Ollama, Open WebUI, or ComfyUI in Docker with full GPU acceleration
  • Fix the same error on native Linux, rootless Docker, and WSL2

Honest take: If nvidia-smi works on your host but not inside a container, you’re missing the NVIDIA Container Toolkit. It’s a five-minute install, and the single most common mistake is forgetting to restart the Docker daemon afterward.

What the error actually means

You run something like:

docker run --rm --gpus all ollama/ollama nvidia-smi

and Docker answers:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

Read the two clues. The capabilities: [[gpu]] part is your --gpus all flag being parsed correctly — Docker understood you want a GPU. The empty "" where a driver name should be is the real signal: Docker looked for a device driver that can satisfy a GPU request and found nothing registered.

That driver is the NVIDIA Container Toolkit. It’s the bridge between the Docker daemon and your host’s NVIDIA driver — it injects the right device nodes and libraries into the container at launch. Your GPU driver (the thing nvidia-smi talks to) and the container toolkit are two separate packages. You almost certainly have the first and are missing the second.

This error has nothing to do with your model, your image, or CUDA out of memory. Those are a different class of problem. This is purely “Docker can’t hand a GPU to the container.”

Step 0: confirm the host driver works first

Before touching anything, prove the GPU is healthy on the host. Run, on the host (not in a container):

nvidia-smi

If that prints your card, driver version, and CUDA version, your GPU driver is fine and this is a container-toolkit problem — skip to Step 1.

If nvidia-smi itself fails with “command not found” or “couldn’t communicate with the NVIDIA driver,” stop. No container fix will help until the host driver works. Install the driver (sudo ubuntu-drivers install on Ubuntu, or NVIDIA’s .run installer), reboot, and confirm nvidia-smi before continuing.

This one check saves the most time. People spend an hour reconfiguring Docker when the actual problem is a driver that never loaded after a kernel update.

Step 1: install the NVIDIA Container Toolkit (the 90% fix)

On Ubuntu/Debian, add NVIDIA’s repository and install. These are the canonical commands from NVIDIA’s install guide, and they haven’t changed in 2026:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

As of this writing the current release is 1.19.1 (published May 21, 2026). If you want to pin an exact version, apt-get install -y nvidia-container-toolkit=1.19.1-1 works once the repo is added.

Step 2: configure Docker to use the runtime — then RESTART

Installing the package isn’t enough. You have to register the runtime with Docker and restart the daemon. This is the step people skip:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The nvidia-ctk runtime configure --runtime=docker command edits /etc/docker/daemon.json and adds the nvidia runtime. The restart is mandatory — Docker reads daemon.json only at startup, so a freshly configured runtime is invisible until you restart. If you configured the toolkit yesterday and it’s still failing, this is almost certainly why.

Confirm the runtime is registered:

docker info | grep -i runtime

You should see nvidia listed under Runtimes. If you don’t, daemon.json wasn’t updated — check cat /etc/docker/daemon.json for an nvidia block.

Step 3: verify with a clean test

Don’t debug with your 40GB model image. Use a tiny CUDA base image whose only job is to run nvidia-smi:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu24.04 nvidia-smi

If this prints the same table you saw on the host, GPU passthrough works and the problem was never your application container. Move on to running Ollama or ComfyUI.

If this still fails after Steps 1–2, you’re in the 10% — keep reading for WSL2 and rootless.

The WSL2 trap that wastes an afternoon

Running Docker inside WSL2 on Windows adds one rule that reverses your instinct: do not install the Linux NVIDIA driver inside the WSL2 distro. The GPU driver lives on the Windows host. WSL2 exposes it through a special /usr/lib/wsl/lib/ path, and installing a normal Linux .run or apt driver inside the distro overwrites that and breaks passthrough entirely.

The correct WSL2 setup:

  1. Windows host: install NVIDIA’s driver with WSL support. CUDA in WSL2 requires driver 470.76 or later; for current cards use the latest Game Ready / Studio driver.
  2. Inside the WSL2 distro: install only the container toolkit, not the GPU driver: 
    sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
  3. If you use Docker Desktop (not Docker installed directly in the distro), enable “Use the WSL 2 based engine” in Settings. Docker Desktop 4.34+ can proxy the GPU without a toolkit inside the distro, but installing the toolkit anyway is the more reliable path.

Confirm nvidia-smi runs inside the WSL2 distro before you touch Docker. If it doesn’t, the Windows host driver is the problem, not Docker. This is the same host-first principle as our Ollama GPU troubleshooting guide, which covers the non-Docker Windows and WSL2 paths in more depth.

Rootless Docker: the extra config nobody documents

If you run Docker rootless (daemon under your user account, not root), the standard runtime configure isn’t sufficient. Rootless containers can’t manage cgroup devices, and the toolkit trips over it. Two edits:

Configure the rootless runtime explicitly:

nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json
systemctl --user restart docker

Then set no-cgroups = true in /etc/nvidia-container-runtime/config.toml:

[nvidia-container-cli]
no-cgroups = true

Toolkit 1.19.0+ improved this specifically — it added support for running containers as a user without explicit device-node access, so rootless setups no longer need you to hand-specify extra groups. If you’re on an older toolkit and hitting permission errors on /dev/nvidia* inside rootless containers, upgrading to 1.19.x is the cleanest fix.

The modern way: CDI (--device nvidia.com/gpu=all)

The --gpus all flag is the legacy path. The Container Device Interface (CDI) is the newer, more explicit mechanism, and it’s what NVIDIA now recommends for anything beyond a single default GPU. Generate the spec once:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Then reference devices by name instead of the --gpus flag:

docker run --rm --device nvidia.com/gpu=all \
  nvidia/cuda:12.4.1-base-ubuntu24.04 nvidia-smi

CDI shines on multi-GPU boxes — --device nvidia.com/gpu=0 and --device nvidia.com/gpu=1 pin containers to specific cards without the NVIDIA_VISIBLE_DEVICES dance. One caveat: regenerate the spec after a driver upgrade (nvidia-ctk cdi generate again), because the CDI file hard-codes device and library paths that a driver update can move. A stale /etc/cdi/nvidia.yaml produces confusing “device not found” errors after an otherwise-routine apt upgrade.

Now run your actual AI stack

With passthrough working, here’s how the common local-AI containers ask for the GPU.

Ollama — the flag is --gpus=all:

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama ollama/ollama

One gotcha worth knowing: when serving 8B-class models in a constrained container, set OLLAMA_NUM_PARALLEL=1 (as an -e env var) to avoid the runner spawning parallel slots that push you into a CUDA out-of-memory crash.

Docker Compose — this is where a second, subtler error hides. There are two ways to request a GPU in Compose, and mixing them up breaks things. The modern form:

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  ollama:

The deploy.resources.reservations.devices block is the Compose equivalent of --gpus all. If your Compose file predates this syntax and uses the old runtime: nvidia key, it still works only if nvidia is set as a runtime in daemon.json — which is exactly what nvidia-ctk runtime configure did in Step 2. If you skipped that and relied on runtime: nvidia, you’ll get the same “could not select device driver” error from Compose. Prefer the deploy block; it doesn’t depend on the default runtime.

ComfyUI and vLLM follow the identical pattern — --gpus all on docker run, or the deploy block in Compose. For a production ComfyUI container with systemd and reverse-proxy access, our ComfyUI Linux production setup guide covers the full stack. If vLLM specifically won’t start even with the GPU visible, that’s a separate engine-init problem — see our vLLM startup-error guide.

When it’s still failing after all of this

A short checklist for the stubborn 5%:

  • Secure Boot can block the NVIDIA kernel module from loading, so nvidia-smi fails on the host after a kernel update. Either sign the module or disable Secure Boot.
  • Kernel updated, driver didn’t rebuild — DKMS should handle this, but if nvidia-smi broke right after apt upgrade, reinstall the driver so the module rebuilds against the new kernel.
  • Docker snap package on Ubuntu is confined and often can’t see the toolkit. Uninstall the snap and install Docker from Docker’s official apt repo instead.
  • /etc/docker/daemon.json is invalid JSON — a stray comma from hand-editing means Docker silently ignores the file (and your runtime). Validate it, then restart.

If none of that lands and you just need a GPU now for a one-off job, renting is faster than debugging: a cloud instance on RunPod ships with the container toolkit pre-configured, so --gpus all works out of the box. But for a permanent local setup, the Steps 1–3 install is the real fix — and once it’s done, it stays done. A used RTX 3090 remains the value pick for a 24GB local box; see our used 3090 breakdown for current pricing.

Building a local AI coding workflow on top of this container? Our sister site aicoderscope.com covers the editor-side tooling, and aifoss.dev tracks the open-source self-hosting stack.

FAQ

Do I need to install CUDA on the host to run GPU containers? No. You need the NVIDIA driver on the host (what nvidia-smi uses) and the container toolkit. The CUDA toolkit lives inside the container image (e.g. nvidia/cuda:12.4.1-...). Installing CUDA on the host is unnecessary for containerized workloads.

Why does nvidia-smi work but Docker still says “could not select device driver”? Because those use different components. nvidia-smi talks to the host GPU driver, which is installed. Docker needs the container toolkit to expose that GPU to containers, and that’s the missing piece. Install it (Step 1), configure the runtime, and restart Docker.

I installed the toolkit and it still fails. What did I miss? Almost always the sudo systemctl restart docker in Step 2. Docker reads its runtime config only at startup. Run docker info | grep -i runtime — if nvidia isn’t listed, the config didn’t take or Docker wasn’t restarted.

Does this work the same on WSL2? Yes, with one rule: install the container toolkit inside the WSL2 distro, but install the GPU driver only on the Windows host (470.76+). Installing a Linux driver inside WSL2 breaks passthrough.

--gpus all vs --device nvidia.com/gpu=all — which should I use? Both work. --gpus all is the legacy flag; CDI (--device nvidia.com/gpu=all, after nvidia-ctk cdi generate) is the newer, more explicit approach and is better for pinning specific GPUs on multi-card systems. Regenerate the CDI spec after any driver upgrade.

Sources

Last updated July 4, 2026. Package versions and driver requirements change; verify current releases before installing.

Was this article helpful?