Local AI Crawling on Windows? Shared GPU Memory Is Why — the NVIDIA Sysmem Fallback Fix (2026)

By RunAIHome Team · 12 min read
nvidiawindowstroubleshootingvramlocal-llmcomfyui

TL;DR: Since driver 536.40 (June 29, 2023), NVIDIA’s Windows driver no longer throws an out-of-memory error when a CUDA app overfills your VRAM — it silently spills the overflow into system RAM (“shared GPU memory”) and your tokens/sec or it/s falls off a cliff. No error, no crash, just a 5–10× slowdown. The fix is one setting: NVIDIA Control Panel → Manage 3D Settings → CUDA - Sysmem Fallback Policy → Prefer No Sysmem Fallback.

What you’ll be able to do after this guide:

  • Confirm in under two minutes — from Task Manager’s “Shared GPU memory” counter and ollama ps — whether spill is what’s eating your speed.
  • Turn the fallback off globally or per-app (the per-app route is safer if you also game on the machine), and know why the app must be restarted afterward.
  • Read the loud CUDA OOM error you’ll get instead as the useful signal it is, and right-size the model, quant, or context so neither failure mode comes back.

Honest take: for LLMs and image generation, a crash you can see beats a crawl you can’t. Flip the policy to Prefer No Sysmem Fallback for your AI apps, let it fail loudly, then fix the actual problem — a model or context that doesn’t fit your card.

Here’s the pattern, and if you’re on Windows with an NVIDIA card you’ve probably lived it: a model or workflow that ran fine yesterday now generates at a tenth of its normal speed. Ollama still answers. ComfyUI still finishes the image. Nothing errored. The GPU fans aren’t even trying. You added a bigger LoRA, or raised the context window, or the new model is 800 MB larger than the old one — and everything quietly turned to sludge.

That’s not a broken install, and it’s not thermal throttling. It’s your VRAM overflowing into system RAM through a driver feature most people don’t know exists — and the driver considers this a success.

What NVIDIA changed, and why your app crawls instead of crashing

Before mid-2023, a CUDA application that asked for more memory than your card had got a hard CUDA out of memory error. In driver 536.40, released June 29, 2023, NVIDIA introduced system memory fallback: when GPU memory is exhausted, the driver transparently backs new allocations with system RAM instead of failing (NVIDIA support article 5490, TechPowerUp release notes). NVIDIA built it so Stable Diffusion would stop crashing on the 6–8 GB cards most people own (VideoCardz coverage).

The RAM it borrows is what Windows calls shared GPU memory — a WDDM budget of up to roughly half your installed system RAM that the OS lets the GPU address (Microsoft DirectX team explainer, NVIDIA developer forums). With 32 GB of RAM, Task Manager will show a 16 GB shared pool sitting next to your card’s dedicated VRAM. It looks like free real estate. It is not.

The bandwidth math: why spill is a 10x problem, not a 10% problem

Model weights and KV cache that land in shared memory don’t get magically faster RAM — every access crosses the PCIe bus. Compare what the GPU was designed to read from against what it’s actually reading from once spill starts:

Memory the GPU is readingBandwidthSource
RTX 4090 GDDR6X (dedicated VRAM)1,008 GB/sTechPowerUp
RTX 3090 GDDR6X (dedicated VRAM)936 GB/sTechPowerUp GPU database
Shared GPU memory over PCIe 4.0 x16~32 GB/s per directionTechPowerUp PCIe scaling review

That’s a ~30× bandwidth gap on paper between an RTX 4090’s VRAM and the PCIe 4.0 link it spills over (1,008 ÷ 32). LLM decode speed is memory-bandwidth-bound — the GPU re-reads the active weights for every single token — so even a modest spill poisons the whole pipeline: the compute stalls waiting on the slow slice. Community threads have reported the real-world damage since the feature shipped, with generations running several times to roughly an order of magnitude slower once spill kicks in (oobabooga text-generation-webui discussion, HardForum thread). One widely-shared test on an 8 GB card found disabling the fallback delivered a large SDXL speedup by forcing the workload to be sized to actual VRAM (Furkan Gözükara, tested on 8 GB GPU).

The insidious part: it’s not binary. Spill 500 MB of a 20 GB working set and you lose some speed; spill 4 GB and you’re running at PCIe pace. That’s why the slowdown feels random — it scales with exactly how far past your VRAM you’ve crept.

The 2-minute diagnosis

1. Task Manager, Performance tab, GPU. Watch two counters while your model loads and generates: Dedicated GPU memory and Shared GPU memory. Dedicated pinned at the top of its range (say, 23.5/24.0 GB) while shared climbs from ~0.1 GB to multiple GB during generation is the fingerprint. A large shared budget (half your RAM) is normal; shared usage growing under AI load is the problem (Microsoft DirectX blog on what the counters mean).

2. For Ollama: check the processor split, then distrust it slightly.

$ ollama ps
NAME          ID            SIZE     PROCESSOR          UNTIL
qwen3:32b     e5c92f7e8fd4  22 GB    27%/73% CPU/GPU    4 minutes from now

A CPU/GPU split means Ollama itself decided the model doesn’t fit and parked layers in system RAM — that’s Ollama’s own scheduler doing partial offload, covered in Ollama Not Using GPU?. But 100% GPU does not guarantee you’re safe on Windows: if the driver spilled part of that “GPU” allocation to shared memory, Ollama has no idea — the layers report as GPU-resident while physically living in RAM behind the PCIe bus (ollama issue #7584). A modded 22 GB RTX 2080 Ti owner watched allocations stop at 19–20 GB with the remaining 2–3 GB forced into shared memory, dragging performance despite “full GPU” loading (ollama issue #10982).

3. Cross-check with nvidia-smi. If Task Manager shows shared usage climbing while nvidia-smi reports VRAM at 98–100% of capacity, you’ve confirmed it: the card is full and the overflow went to RAM.

The fix: Prefer No Sysmem Fallback

The setting shipped alongside the feature, and NVIDIA documents it in the same support article (NVIDIA article 5490):

  1. Open NVIDIA Control Panel (right-click desktop, or search the Start menu).
  2. Go to Manage 3D Settings.
  3. Find CUDA - Sysmem Fallback Policy. Three options: Driver Default, Prefer Sysmem Fallback, Prefer No Sysmem Fallback.
  4. Select Prefer No Sysmem Fallback.
  5. Restart the AI application — the policy is read when the CUDA context is created, so a running Ollama service or ComfyUI session keeps the old behavior until relaunched. NVIDIA’s article calls the restart out explicitly.

Global vs per-program: the Global Settings tab changes every CUDA app on the machine. If the box also handles gaming or video work, use the Program Settings tab instead — click Add, browse to the executable, and set the policy for just that program. For Python-based tools like ComfyUI, point it at the actual interpreter that runs the workload, e.g. ComfyUI\venv\Scripts\python.exe — not a launcher script (walkthrough with screenshots). For Ollama on Windows, add ollama.exe — and note some users have reported the runner behaving as if the policy didn’t stick, so verify with the Task Manager check afterward rather than assuming (ollama issue #16725).

There’s one setting-adjacent trap worth knowing: “Prefer” is in the name because it’s a preference, not a hard guarantee — but in practice, with the policy set, allocations that would have spilled now fail fast with a visible error.

What breaks next (on purpose), and how to fix that properly

With fallback off, the workload that was silently spilling now dies with CUDA out of memory — the classic, honest failure. Good. Now you can fix the real problem instead of eating a hidden 10×. The complete triage lives in CUDA Out of Memory on Local AI? Every Fix That Works, but the short version by app:

Ollama: drop one quant level (Q4_K_M instead of Q5/Q6) or trim num_ctx — KV cache grows linearly with context and is usually what pushed you over. Why your tokens/sec dropped covers the measurement loop, and if the model genuinely can’t fit, Ollama’s own CPU offload is the correct slow path because at least it’s visible in ollama ps. Running 70B-class models on a 24 GB card is its own discipline — see Run a 70B Model on a 24GB GPU.

ComfyUI: leave a buffer so the driver never has a reason to spill — launch with --reserve-vram 1.5 to hold back headroom for the OS and browser, and add --disable-smart-memory if models stack up in VRAM across runs (Comfy-Org issue #6314, ComfyUI troubleshooting docs). SDXL VAE decode spikes are the classic spill trigger on 8–12 GB cards.

LM Studio: the app ships its own pre-flight called Model Loading Guardrails (Settings → strictness levels from Relaxed to Strict) that estimates whether a load will overwhelm your memory and blocks it first (lmstudio bug tracker #128). Guardrails plus No-Fallback is the belt-and-suspenders setup; if loads that used to work start getting blocked, tune the GPU offload slider — LM Studio Not Using GPU? walks through it.

Everything at once: your VRAM budget is finite and shared. A browser with hardware acceleration takes a few hundred MB; Windows itself takes some; two AI apps at once take it all. Close what you’re not using before assuming the model got slower. And note this is a Windows driver behavior — the same overflow inside WSL2 goes through a different path, covered in WSL 3 GPU Passthrough for Local AI.

The one case to leave fallback ON

The feature isn’t dumb — it’s mis-targeted for interactive use. Leave Driver Default (or explicitly Prefer Sysmem Fallback) when finishing matters more than speed: an overnight batch render, a one-off upscale that needs 2 GB more than you have, a model conversion job. NVIDIA built the feature precisely so a 6 GB card could complete a job that would otherwise crash, and for unattended work “slow but done by morning” beats “crashed at 1 a.m.” If that describes most of your workload, per-program settings let you have both: no-fallback for Ollama, fallback allowed for the batch tool.

And if you’re constantly 4+ GB past your card, no driver policy fixes that — you’re shopping. The honest options are a used RTX 3090 (24 GB, 936 GB/s — still the value play; full breakdown), a new RTX 5060 Ti 16GB if your models fit in 16 GB (check which models fit which VRAM tier), or renting the occasional big job on RunPod by the hour instead of owning VRAM you need twice a month.

FAQ

Does this affect Linux? No. System memory fallback is a Windows driver feature; on Linux, CUDA allocations past VRAM fail with OOM as they always have (llama.cpp/Ollama CPU offload still exists there, but that’s explicit layer placement, not silent driver spill).

Why does Task Manager show 16 GB of shared GPU memory on my 8 GB card? Did a driver give me free VRAM? No — that’s the WDDM budget: Windows allows the GPU to address up to about half your system RAM (Microsoft’s explainer). It’s an accounting ceiling, not fast memory. Usage of it under AI load is what you’re watching for.

I set Prefer No Sysmem Fallback and nothing changed. Restart the app — the policy applies when the CUDA context starts, and a resident Ollama service or open ComfyUI tab keeps the old policy until relaunched. If you set it per-program, confirm you added the executable that actually runs CUDA (the venv python.exe, not a .bat launcher).

Will games get slower with the global setting? Games don’t use CUDA sysmem fallback — this policy governs CUDA allocations, not Direct3D. Still, per-program settings for your AI apps are the cleaner change and leave everything else at driver default.

Is shared GPU memory the same as Ollama offloading layers to CPU? Different mechanisms, same physics. Ollama’s partial offload is deliberate and visible (ollama ps shows the CPU/GPU split); driver spill is invisible to the app. Both are slow because weights end up behind a bus that’s ~30× slower than VRAM. The difference is you can see one of them — which is exactly why this fix makes the invisible one impossible.

Sources

Last updated July 30, 2026. Driver behavior and control panel labels are current as of the 2026 driver branches; verify against NVIDIA’s support article if a future driver renames the setting.

  • RTX 3090 — 24 GB / 936 GB/s used-market value pick when you’re chronically past your VRAM
  • RTX 5060 Ti 16GB — the affordable new-card exit from 8–12 GB spill purgatory

Was this article helpful?