TL;DR: Failed to initialize NVML: Driver/library version mismatch almost always means an apt upgrade swapped your NVIDIA user-space libraries while the old kernel module was still loaded — you can fix it by reloading the modules, no reboot needed. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver is the worse one: the kernel module isn’t loaded at all, usually a DKMS build failure or Secure Boot block.

What you’ll be able to do after this:

• Tell the two errors apart in 10 seconds and know which fix applies
• Recover a running headless AI server from the “mismatch” error without rebooting (and without killing your in-flight training job if you’re careful)
• Stop unattended-upgrades from silently breaking your GPU stack again next week

Honest take: If nvidia-smi says mismatch, don’t reboot on reflex — reload the kernel modules and you keep your uptime. If it says couldn’t communicate, stop guessing and read dmesg | grep NVRM; the module either failed to build or Secure Boot refused to load it, and both tell you exactly in the log.

These two errors dominate the “my Ollama/ComfyUI/vLLM suddenly can’t see the GPU” support threads, and the reason is almost never the AI tool — it’s the driver layer underneath. This is the companion piece to the Docker GPU passthrough fix and Ollama not using the GPU: before you debug the app, confirm nvidia-smi itself works.

First, which error do you have?

Run nvidia-smi. You’ll get one of two failures, and they have completely different root causes.

Error A — the mismatch:

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.95

The driver is loaded. The user-space library and the loaded kernel module just disagree on their version. This is the recoverable one.

Error B — no communication:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

The kernel module isn’t loaded at all — or the GPU fell off the PCIe bus. This one needs real diagnosis.

The single command that separates them:

lsmod | grep nvidia

If that prints several lines (nvidia_uvm, nvidia_drm, nvidia_modeset, nvidia), the module is loaded and you have Error A. If it prints nothing, the module isn’t loaded and you have Error B.

Error A: “Driver/library version mismatch”

What actually happened

You (or unattended-upgrades) ran an update that pulled a new NVIDIA driver package. The new files landed on disk — including the new libnvidia-ml.so (for example libnvidia-ml.so.580.95.05 on the current 580 production branch) — but the kernel module already loaded in RAM is still the old version. User-space now talks to a library that expects a newer kernel module than the one running. NVML refuses to initialize. (NVIDIA Developer Forums, Exxact)

You can see the split directly. The loaded kernel module version:

cat /proc/driver/nvidia/version

versus what’s now installed on disk:

dpkg -l | grep nvidia-driver     # Debian/Ubuntu
# or
modinfo nvidia | grep ^version

When those two numbers differ, that’s your confirmation. A reboot would fix it (it reloads the new module) — but on a headless AI server you usually don’t want to reboot, and you don’t have to.

The no-reboot fix

Reload the kernel modules so the running module matches the new library. Unload them in dependency order (top-down), then let modprobe reload:

sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia
sudo modprobe nvidia
nvidia-smi

nvidia-smi should now succeed, reporting the new version. This resets the driver environment in place and aligns it with the NVML library. (Exxact, NVIDIA Developer Forums)

“rmmod: ERROR: Module nvidia is in use”

This is the common wrinkle. Something is holding the module — an inference server, a training run, nvidia-persistenced, or your display manager on a desktop. You cannot unload a module that’s in use. Find the culprits:

sudo lsof /dev/nvidia*

That lists every process with a GPU device open. On a headless server this is usually your own workload: Ollama’s ollama process, a python training script, a vllm server, ComfyUI. Stop them cleanly (or sudo kill <PID>), then retry the rmmod sequence.

Two processes that people forget:

• nvidia-persistenced — the persistence daemon holds the module open. Stop it first: sudo systemctl stop nvidia-persistenced, do the reload, then start it again.
• Your display manager (desktop only) — if you’re on a machine with a monitor, X or Wayland has the GPU. Drop to a text console (sudo systemctl isolate multi-user.target), reload, then sudo systemctl isolate graphical.target.

The hard truth: if you have a training run in progress with hours of state in VRAM, there is no way to reload the driver without losing it — the module is in use by definition. You either wait for the job to finish or you reboot. Plan driver upgrades for when the box is idle. (If you’re stuck and need to keep a workload running while you rebuild the host driver stack, spinning up a RunPod instance for the interim is cheaper than the lost hours.)

If the modules still won’t cooperate

Occasionally nvidia_drm won’t unload because of modeset. If you use the modeset/DRM path (common with Wayland), check fuser -v /dev/nvidia* for stragglers. When a clean unload is genuinely impossible and the box is expendable of its current state, a reboot is the honest fallback — it’s not a defeat, it’s the correct tool when the module is pinned.

Error B: “couldn’t communicate with the NVIDIA driver”

lsmod | grep nvidia returned nothing. The module isn’t loaded. Now you diagnose why. The kernel log tells you:

sudo dmesg | grep -i nvrm
sudo dmesg | grep -i nvidia

Cause 1 — DKMS didn’t rebuild after a kernel update (most common)

The NVIDIA driver ships as a DKMS module: when apt installs a new kernel, a post-install hook is supposed to recompile the NVIDIA module against it. If that compile fails — missing headers, no compiler, a kernel too new for the driver branch — you boot into the new kernel with no NVIDIA module. Check DKMS state:

dkms status

If you see added instead of installed next to the nvidia entry, the module was registered but never built for your running kernel. (oneuptime, Arch Wiki)

The fix is to install the matching headers and rebuild:

sudo apt install linux-headers-$(uname -r)
sudo dkms autoinstall
sudo modprobe nvidia
nvidia-smi

linux-headers-$(uname -r) is the step people skip — DKMS cannot compile a kernel module without the headers for the currently running kernel, and a fresh kernel often ships without them. If autoinstall still fails, build the specific module by hand to see the compiler error:

sudo dkms install -m nvidia -v 580.95.05 -k $(uname -r)

(Substitute your actual driver version — dkms status prints it.) Read the build log it points to; a missing gcc or a header/kernel version skew is the usual reason.

Cause 2 — Secure Boot is rejecting an unsigned module

If Secure Boot is on, the kernel refuses to load a module it can’t verify. The module builds fine, but never loads — and dmesg shows something like Loading of unsigned module is rejected. Check:

mokutil --sb-state
modinfo nvidia | grep -i sig

If Secure Boot is enabled and the module is unsigned, you have two options: enroll a Machine Owner Key (MOK) so the signed module is trusted (Ubuntu prompts for this at driver install — you must complete the blue MOK enrollment screen on the next reboot), or disable Secure Boot in firmware. On a headless server that never runs an unsigned bootloader anyway, disabling Secure Boot is the pragmatic choice. (NVIDIA Developer Forums)

Cause 3 — nouveau grabbed the GPU

The open-source nouveau driver can claim the card before the NVIDIA module. Check:

lsmod | grep nouveau

If it’s loaded, the proprietary driver install should have blacklisted it. Confirm the blacklist exists (/etc/modprobe.d/blacklist-nouveau.conf), regenerate the initramfs (sudo update-initramfs -u), and reboot once so nouveau is excluded from boot.

Cause 4 — the GPU fell off the bus

If dmesg shows NVRM: Xid errors or GPU has fallen off the bus, this isn’t software. It’s power, thermals, or a PCIe seating/riser problem — common on multi-GPU rigs with cheap risers. Confirm the card is even visible:

lspci | grep -i nvidia

If lspci doesn’t list the GPU, no driver fix will help — reseat the card, check the PCIe power cables, and review your PSU headroom (see PSU sizing for AI workstations).

Stop it from happening again

The mismatch error is self-inflicted 90% of the time: unattended-upgrades pulling a new driver on a running box. Two durable fixes.

Option 1 — exclude NVIDIA packages from automatic upgrades. Edit /etc/apt/apt.conf.d/50unattended-upgrades and add a blacklist:

Unattended-Upgrade::Package-Blacklist {
    "nvidia-";
    "libnvidia-";
    "cuda-";
};

Or turn unattended upgrades off entirely with sudo dpkg-reconfigure unattended-upgrades and choose “No”. (NVIDIA Developer Forums)

Option 2 — pin the driver version. This freezes it until you deliberately upgrade during a maintenance window:

sudo apt-mark hold nvidia-driver-580 libnvidia-compute-580

Either way, the rule is: you decide when the driver changes, and you reboot (or reload modules) immediately after. Never let a background job swap the driver under a running kernel.

For containerized setups, note the same trap bites Docker: when the host driver upgrades, running containers keep the old library reference and throw the mismatch inside the container too. Restart the container after any host driver change — see the Docker GPU passthrough guide for the full container-side story.

Quick reference

Symptom	Root cause	Fastest fix
NVML: Driver/library version mismatch, lsmod shows nvidia	Old module loaded, new libs on disk	rmmod the 4 modules → modprobe nvidia
rmmod: Module nvidia is in use	A process/daemon holds the GPU	lsof /dev/nvidia*, stop it, stop nvidia-persistenced, retry
couldn't communicate, lsmod empty, dkms status = added	Module not built for new kernel	apt install linux-headers-$(uname -r) → dkms autoinstall
couldn't communicate, dmesg = unsigned module rejected	Secure Boot	mokutil --sb-state, enroll MOK or disable Secure Boot
couldn't communicate, nouveau loaded	Nouveau not blacklisted	Blacklist nouveau, update-initramfs -u, reboot
GPU has fallen off the bus / not in lspci	Hardware: power/PCIe/thermals	Reseat card, check PSU/risers

Once nvidia-smi reports your GPU again, your AI stack comes back on its own — but if Ollama or ComfyUI still runs on CPU, that’s a separate issue covered in Ollama not using the GPU and CUDA out of memory fixes. Coding on the same box? aicoderscope.com covers the local-LLM coding stack, and aifoss.dev covers the self-hosting side.

FAQ

Do I really not need to reboot for the mismatch error? Correct — for Driver/library version mismatch, reloading the kernel modules (rmmod the four, then modprobe nvidia) is enough, as long as you can free the module. A reboot does the same thing but costs you uptime. For couldn't communicate, you often do end up rebooting once (after fixing headers/DKMS or blacklisting nouveau), because the module has to load cleanly at boot.

Why does nvidia-smi show a version but still say mismatch? The version printed in the mismatch message (NVML library version: 580.95) is the user-space library version — the new one from your upgrade. The loaded kernel module is older; cat /proc/driver/nvidia/version shows that older one. The gap between them is the whole problem.

Will reloading the modules interrupt my running model? Yes — any process using the GPU must be stopped before the module can unload. There’s no way around it; the module is “in use” precisely because your model is loaded in VRAM. Schedule driver reloads for idle windows.

Is this an NVIDIA-only problem? The specifics (NVML, nvidia_uvm) are NVIDIA. AMD ROCm has its own kernel-module-versus-userspace failure modes after a kfd/driver upgrade, but the principle — new libs, old loaded module — is the same. See the ROCm setup guide.

I upgraded on purpose and now it’s broken — did I do something wrong? No. A deliberate driver upgrade lands new files without reloading the running module; that’s expected. The correct habit is to reboot (or reload modules) right after any driver upgrade, before you start work. The error only surprises you when a background upgrade does it silently.

Sources

• Failed to initialize NVML: Driver/library version mismatch — NVIDIA Developer Forums
• How to fix NVML: Driver/library version mismatch without rebooting — NVIDIA Developer Forums
• Ubuntu “unattended-upgrades” leads to NVML mismatch — NVIDIA Developer Forums
• GPU Troubleshooting Guide: Resolving Driver/Library Version Mismatch Errors — Exxact
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver — NVIDIA Developer Forums
• How to Troubleshoot NVIDIA Driver Issues After Kernel Update on Ubuntu — OneUptime
• Dynamic Kernel Module Support (DKMS) — Arch Wiki
• nvidia-smi is returning NVML: Driver/library version mismatch — Hyperstack

Last updated July 11, 2026. Driver version numbers (580 branch) reflect the current production branch at time of writing; substitute your installed version. Commands target Debian/Ubuntu; adapt package names for Fedora/RHEL.