NVMe KV Cache Offloading for Local LLMs in 2026: What Spilling Context to Your SSD Actually Buys a 24GB GPU

By RunAIHome Team · 14 min read
kv-cachenvmevllmllama-cpplocal-llmgpulong-context

TL;DR: NVMe KV cache offloading — the technique NVIDIA standardized at CES 2026 with ICMSP — spills attention context from GPU VRAM to CPU RAM and then to your SSD. On a home rig it will not make decoding faster or raise your maximum context, but it kills the single most painful long-context tax: re-processing 100K tokens of prompt every time you return to a conversation.

What you’ll be able to do after this guide:

  • Calculate exactly how much VRAM your KV cache eats per token, so you know whether context — not model weights — is what’s actually limiting you
  • Set up a GPU → RAM → NVMe cache tier with vLLM + LMCache, or persistent per-session KV saves with llama.cpp’s --slot-save-path
  • Recognize the two workloads where NVMe offloading backfires, before you spend an evening configuring it

Honest take: If you run multi-turn agent sessions or return to long documents repeatedly, KV offloading to a fast NVMe drive is free capacity you already own. If you mostly do single-shot chat under 2K tokens of context, skip it — the I/O adds latency and saves you nothing.

This is not the model-loading article

The site already covers why your NVMe drive matters for model loading — that’s a one-time, sequential read when the weights come off disk. This article covers something different: the KV cache, the memory that grows while the model runs.

Every token the model processes leaves behind key and value tensors — the context the attention mechanism re-reads on every subsequent token. Unlike model weights, which are a fixed cost, the KV cache grows linearly with context length and with the number of parallel sessions. It lives in VRAM by default, in the same pool as your weights. Long context on a consumer GPU is a three-way fight between weights, KV, and headroom — and KV is the part that offloading can move.

The math: what context actually costs in VRAM

The KV cache size per token is fixed by the model architecture:

bytes per token = 2 (K and V) × layers × KV heads × head dim × bytes per value

Computed from published model architectures at FP16 (these are derived numbers — the formula is standard, see the sources):

ModelLayers × KV heads × head dimKV per tokenKV at 32K ctxKV at 128K ctx
Llama 3.1 8B32 × 8 × 1280.125 MB4 GB16 GB
Llama 3.3 70B80 × 8 × 1280.31 MB10 GB40 GB

That last column is the story. A 70B model’s KV cache at 128K context is 40 GB — nearly as large as the ~42.5 GB of Q4_K_M weights themselves, and bigger than any consumer GPU’s VRAM on its own. Even the 8B model at full 128K context needs 16 GB of KV — two-thirds of a used RTX 3090’s 24 GB (~$1,252 average in July 2026 per Best Value GPU) before a single weight is loaded.

This is why your 30B-A3B coding model that “fits comfortably” in 24 GB at 8K context OOMs the moment you push the context up: the weights didn’t grow, the KV did. Quantizing the KV cache to q8_0 halves these numbers and flash attention trims the overhead — both covered in the Ollama tuning guide — but quantization only delays the wall. Past a certain context, the KV has to go somewhere that isn’t VRAM.

Why this became datacenter news in 2026

At CES 2026, NVIDIA announced the Inference Context Memory Storage Platform (ICMSP) — a standardized way to extend GPU KV cache through CPU DRAM out to NVMe SSDs, managed by BlueField-4 DPUs so the GPU’s compute units don’t stall on PCIe I/O. NVIDIA claims up to 5× better power efficiency and up to 5× higher tokens per second versus traditional storage approaches (vendor numbers, no independent benchmarks yet), with BlueField-4 shipping in the second half of 2026 and Dell, HPE, Pure Storage, VAST Data, WEKA and others building products on it.

The research side moved too. DUAL-BLADE (arXiv, April 29, 2026 — Sogang University, Florida State, and Samsung) targets exactly the home-lab-shaped problem: file-based KV offloading thrashes the kernel page cache under memory pressure. Their fix — mapping KV tensors directly to NVMe logical block addresses, bypassing the filesystem — cut prefill latency by up to 33.1% and decode latency by up to 42.4% versus existing file-based offloading on edge hardware, not datacenter racks.

None of that ships in a home-lab tool today. But the direction is unambiguous: the industry decided in 2026 that KV cache belongs on a storage tier, and the software you can run tonight already implements the same idea in cruder form.

What “10× more context” really means (and doesn’t)

The claim that put this technique on r/LocalLLaMA — a single H100 serving 10× more concurrent users with LMCache’s GPU→CPU→SSD tiering, per Spheron’s deployment guide — is about cache capacity across sessions, not about one request’s context window.

Here’s the distinction that most coverage blurs, and the reason to read this section before configuring anything:

  • During active decoding, the KV blocks the model is attending to must be in VRAM. Decode is memory-bandwidth-bound — your GPU reads the model weights plus KV for every token. An NVMe drive at 7.4 GB/s cannot substitute for GDDR6X at 936 GB/s. Offloading does not raise the maximum context a single request can decode against, and it does not make decoding faster.
  • What offloading moves is inactive context: the KV of a session that’s idle, a document you’ll come back to, the 60K-token conversation from this morning. Without a cache tier, that KV gets evicted when VRAM pressure rises — and re-materializing it means re-running prefill over the whole prompt. With a tier, it streams back from RAM or NVMe in seconds.

So the honest pitch for a home lab is: NVMe KV offloading eliminates re-prefill, not VRAM limits. Time-to-first-token is where you feel it. LMCache’s May 2026 benchmark on multi-turn agentic workloads (32 users, ~100K context traces, MI300X) measured 3.0× lower average time-to-first-token and 2.3× more requests served versus GPU-memory-only serving; on their production-scale runs, median TTFT dropped from 18.5 s to 8.0 s and throughput doubled from 0.37 to 0.73 req/s once the cache was warm. Those are server-class GPUs, but the mechanism — skip prefill when the KV already exists somewhere — is identical on a 3090.

Path 1: vLLM + LMCache (the real tiered setup)

vLLM is the one consumer-reachable stack with true GPU→CPU→disk KV tiering, via the LMCache connector. If you already serve models with vLLM, this is two config changes.

Create lmcache.yaml:

chunk_size: 256
local_cpu: true
max_local_cpu_size: 16.0        # GB of system RAM for the hot spill tier
local_disk: "file:///mnt/nvme/lmcache/"
max_local_disk_size: 200.0      # GB of NVMe for the cold tier

Then launch vLLM with the LMCache connector:

LMCACHE_CONFIG_FILE=lmcache.yaml vllm serve Qwen/Qwen2.5-14B-Instruct \
  --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

Expected startup log line confirming the connector loaded:

INFO ... Initializing LMCache connector (kv_role: kv_both)

Key names and the connector flag are current as of LMCache’s 2026 docs and vLLM’s production-stack tutorial — check docs.lmcache.ai if a launch error names an unknown key, because the config schema has changed across releases. Sizing guidance: point local_disk at your fastest NVMe, and give max_local_cpu_size whatever RAM you can spare after the model server itself — system RAM is the cheapest tier you have. If vLLM won’t start at all, that’s a different article.

One caveat for MoE models: LMCache’s April 2026 architecture rework specifically fixed a 10× performance gap for MoE inference with offloading enabled — if you tried this in 2025 with a Mixtral-style model and found it unusable, that result is stale.

Path 2: llama.cpp slot saves (persistent sessions on any GPU)

llama.cpp doesn’t do automatic tiering, but llama-server can save and restore a slot’s entire KV state to disk — which covers the single most common home-lab case: one user, long documents, repeated visits.

The real problem this solves, straight from llama.cpp’s discussion board (#18244): “I have a 950k prompt, how do I not reprocess it every time?” Stock behavior re-prefills the entire prompt each session; a maintainer-declined feature request (#17107) confirms auto-persistence is not coming, so you drive it via the REST API yourself.

Start the server with a save directory:

llama-server -m qwen3-coder-30b-a3b-q4_k_m.gguf -ngl 99 -c 65536 \
  --slot-save-path /mnt/nvme/kv-slots/

After running your long prompt once (with "cache_prompt": true in the completion request), save the slot:

curl -X POST "http://localhost:8080/slots/0?action=save" \
  -H "Content-Type: application/json" -d '{"filename":"project-audit.bin"}'

Expected response:

{"id_slot":0,"filename":"project-audit.bin","n_saved":58312, ...}

n_saved is the token count whose KV state just went to disk. Tomorrow, ?action=restore with the same filename brings the whole context back without recomputing a single token of prefill. One community measurement of the pattern: a 5K-token chat took 9.9 s to re-prefill cold versus 1.4 s to restore from disk — 7× faster, and the gap widens with context length since restore scales with file read speed, not compute. The files are large (hundreds of MB to multiple GB — roughly the KV-per-token math from the table above), which is exactly why the save path should be NVMe.

This pairs naturally with speculative decoding — one trick removes the prefill tax, the other speeds the decode itself.

Path 3 (blocked): Ollama

Ollama has no KV-to-disk capability as of v0.32.5 (July 27, 2026). What it offers instead: KV cache quantization (OLLAMA_KV_CACHE_TYPE=q8_0, requires flash attention) and a runner-level option to keep the KV in system RAM while layers stay on GPU. There’s an open issue (#9750) asking Ollama to prefer offloading layers over KV when both don’t fit — the maintainers’ current split does the reverse — but nothing on the roadmap touches NVMe. If persistent long-context sessions matter to you, this is a genuine reason to run llama-server or vLLM underneath instead of Ollama; your model storage location is the only disk decision Ollama lets you make.

When it backfires

Backend.AI’s April 2026 analysis of KV offloading operating conditions draws the line clearly, and it matches the architecture logic:

  • Short contexts lose. Below ~1K tokens of context, prefill is memory-bound and the offload round-trip costs more than recomputing; their measured turning point is around 2K tokens. At 10K+ tokens, prefill is compute-bound and fetching cached KV over PCIe wins decisively.
  • Decode never speeds up. Decoding is memory-bandwidth-bound, full stop. Any setup that puts actively decoded KV behind PCIe or NVMe degrades tokens/sec. The tier is for parked context, not live context.
  • Slow storage poisons the whole scheme. A SATA SSD at ~550 MB/s turns a 16 GB KV restore into a ~29-second stall; a Samsung 990 Pro at 7,450 MB/s does the same read in roughly 2–3 seconds (our math from the drive specs — sequential-read floor, real restores add overhead). The WD Black SN850X at 7,300 MB/s is the usual ~$40-cheaper alternative; the 990 Pro 2TB tracked at ~$390 in July 2026 (post-NAND-surge pricing — it was a $160 drive in 2024). For KV tiering, unlike one-time model loading, drive speed is felt on every session switch.

And the quiet trap: KV cache files are a plaintext-adjacent record of your context. A saved slot file can be replayed to reconstruct what the model was working on — treat /mnt/nvme/kv-slots/ with the same care as the documents that produced it.

Verdict for a 24GB home lab

Do itSkip it
Multi-turn coding agents re-reading a repo✅ llama-server slot saves or vLLM+LMCache
Long-document Q&A you revisit across days✅ slot save/restore, NVMe-backed
Several family members sharing one GPU server✅ LMCache tiering (more warm sessions per GPU)
Single-shot chat, <2K context❌ pure overhead
Trying to decode 1M-token context on 24GB❌ offloading doesn’t raise the decode ceiling — partial-offload math still rules

The 1M-context models this year made the question urgent — Kimi K3’s weights landed July 27 with a 1M window — but the honest sequence for a home lab is: quantize the KV first, right-size num_ctx second, tier to NVMe third, and rent a big-VRAM pod on RunPod for the genuinely huge one-off jobs (an A100 80GB runs $1.39/hr). If your workflow involves returning to context, the SSD you already own is the cheapest VRAM extension you’ll ever install. For the FOSS tooling side of self-hosted inference servers, aifoss.dev covers the open-source stacks, and aicoderscope.com covers wiring local endpoints into coding agents — the workload where cache reuse pays off hardest.

FAQ

Does NVMe KV offloading let me run longer contexts than my VRAM allows? No. During decoding, the active context’s KV must be in VRAM. Offloading stores inactive KV (idle sessions, earlier turns) so it doesn’t need re-prefilling later. Your max decodable context is still set by VRAM minus weights, times whatever KV quantization buys you.

How much faster is restoring vs re-processing a prompt? Community measurement on llama.cpp: 1.4 s restore vs 9.9 s re-prefill for a 5K-token chat (7×). The gap grows with context length — restore is a sequential disk read; prefill is compute over every token.

What NVMe speed do I need? PCIe 4.0-class sequential reads (7,000+ MB/s — Samsung 990 Pro, WD SN850X) make multi-GB restores feel instant. SATA SSDs (~550 MB/s) technically work but add ~13× the stall on every restore.

Why doesn’t Ollama support this? Ollama’s scheduler manages VRAM by unloading whole models, not by tiering KV. As of v0.32.5 (July 27, 2026) there is no KV-to-disk path; use llama-server or vLLM+LMCache if you need it.

Is this the same thing NVIDIA announced at CES 2026? Same idea, different scale. ICMSP/BlueField-4 does GPU→DRAM→NVMe tiering with dedicated DPU hardware for clusters; LMCache and llama.cpp slot saves do it in software on one machine.

Sources

Last updated July 29, 2026. Prices and specs change; verify current rates before purchasing.

Was this article helpful?