KV Cache VRAM Cost: What 128K Context Actually Costs on a Consumer GPU (2026)
TL;DR: Every token you keep in context costs you VRAM, on top of the weights, and the bill is bigger than most people expect. I read the headers of two real GGUF files today (November 19, 2026) over HTTP range requests and did the arithmetic from the model’s own numbers. For Llama 3.3 70B Instruct Q4_K_M (weights: 42,520,398,816 bytes = 39.60 GiB, measured), the KV cache costs 320 KiB per token at f16 — 2.50 GiB at 8K context, 10.00 GiB at 32K, and 40.00 GiB at the 131,072-token limit the model file itself declares. Switch the KV cache to q8_0 and those become 1.25 / 5.00 / 20.00 GiB; q4_0 quarters them again. That is the difference between a 48 GB card that runs 70B at 8K and one that cannot run 70B at 32K without quantizing the cache.
What you’ll be able to do:
- Compute the exact KV cache footprint of any model from four numbers you can read out of the GGUF header.
- Tell whether your “CUDA out of memory” error is the weights or the context window.
- Pick a cache type (
f16/q8_0/q4_0/f8_e4m3) that fits the card you already own. - Size a card for the context length you actually need, instead of guessing.
Honest take: everything below is arithmetic on measured metadata, plus the published memory sizes from NVIDIA’s own product pages. I did not run a 70B model, I did not measure tok/s, and I make no quality claim about quantized KV caches. The formula is the well-known one (K and V per layer, per token), and every input to it is printed in the table so you can redo the sum for your own model.
The four numbers that decide everything
The KV cache is the part of a running model that holds the attention keys and values for every token in the current context. It grows linearly with the context length, and its size per token is fixed by the architecture:
bytes per token = 2 (keys + values)
x layers
x kv_heads
x head_dim
x bytes per element
Where head_dim = hidden_size / attention_heads, and bytes per element is 2 for f16, 1 for q8_0 (and for the new FP8 cache in llama.cpp v1.0.0), and 0.5 for q4_0.
All four architecture numbers are in the GGUF file. I read them straight out of the file header of bartowski/Llama-3.3-70B-Instruct-GGUF with a small HTTP range request — no download, no guesswork:
| GGUF metadata key | Value |
|---|---|
general.architecture | llama |
general.name | Llama 3.3 70B Instruct |
general.file_type | 15 (MOSTLY_Q4_K_M) |
llama.block_count | 80 layers |
llama.embedding_length | 8192 |
llama.attention.head_count | 64 |
llama.attention.head_count_kv | 8 |
llama.context_length | 131072 |
So head_dim is 8192 / 64 = 128, and at f16:
2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes per token = 320 KiB per token
Multiply by the context length and you have the whole story. Note the 8 KV heads: Llama 3.3 70B uses grouped-query attention, so the KV cache is 8 heads wide while the query side has 64. That is why the number is manageable at all — a 64-head KV cache would be 8x bigger.
What that costs, at four context lengths
Same model, same weights (39.60 GiB), different context and cache type:
| Context | KV at f16 | KV at q8_0 | KV at q4_0 | Weights + KV (f16) |
|---|---|---|---|---|
| 8,192 tokens | 2.50 GiB | 1.25 GiB | 0.63 GiB | 42.10 GiB |
| 32,768 tokens | 10.00 GiB | 5.00 GiB | 2.50 GiB | 49.60 GiB |
| 65,536 tokens | 20.00 GiB | 10.00 GiB | 5.00 GiB | 59.60 GiB |
| 131,072 tokens | 40.00 GiB | 20.00 GiB | 10.00 GiB | 79.60 GiB |
Read the last row again. At the model’s own advertised 128K window, the KV cache at f16 is 40 GiB — as much VRAM as the entire card on a 48 GB machine, spent on keys and values that are mostly padding. The weights are not even the biggest line item anymore.
This is the mechanism behind the “my 24GB card ran the model fine until the context grew” experience: at 8K the KV cache is 2.5 GiB of a 24 GiB card, at 32K it is 10 GiB, and somewhere in between the model no longer loads.
The same math on a 30B MoE
Dense attention is not the only factor. A mixture-of-experts model with a small number of KV heads is much cheaper per token. From unsloth/Qwen3-30B-A3B’s config.json:
| Field | Value |
|---|---|
num_hidden_layers | 48 |
hidden_size | 2048 |
num_attention_heads | 32 |
num_key_value_heads | 4 |
max_position_embeddings | 40960 |
num_experts / num_experts_per_tok | 128 / 8 |
head_dim is 2048 / 32 = 64, so at f16:
2 x 48 x 4 x 64 x 2 bytes = 49,152 bytes per token = 48 KiB per token
And the weights, measured from the Hugging Face tree API for unsloth/Qwen3-30B-A3B-GGUF:
| Quantization | File size (measured) | GiB | + KV at 40,960 tokens (f16) |
|---|---|---|---|
| Q4_K_M | 18,556,686,912 bytes | 17.28 GiB | 19.16 GiB |
| Q5_K_M | 21,725,581,888 bytes | 20.23 GiB | 22.11 GiB |
| Q8_0 | 32,483,932,736 bytes | 30.25 GiB | 32.13 GiB |
A 24 GB card runs Qwen3-30B-A3B at its full 40,960-token context in Q4_K_M with about 5 GiB to spare — and that spare headroom is what the compute buffers, the CUDA context and the runtime need. The 70B model on the same card is not a context problem at all; it does not fit at any context length, because the weights alone are 39.60 GiB.
Choosing the cache type
llama.cpp exposes the cache type per tensor:
$ llama-server -m model.gguf -c 32768 --cache-type-k q8_0 --cache-type-v q8_0
The cache types you will actually see in the wild:
| Cache type | Bytes per element | KV at 32,768 tokens (70B) | When to use it |
|---|---|---|---|
f16 (default) | 2 | 10.00 GiB | You have the VRAM and want the reference behaviour |
q8_0 | 1 | 5.00 GiB | The default choice for a long context on a card that is already tight |
q4_0 | 0.5 | 2.50 GiB | You are squeezing the last gigabyte; more quantization error in the attention path |
f8_e4m3 | 1 | 5.00 GiB | New in llama.cpp v1.0.0 (released November 13, 2026): OCP-format FP8 cache with optional static per-tensor scales |
The v1.0.0 release notes list the new FP8 KV cache under core changes as “FP8 KV caches (-ctk/-ctv f8_e4m3) with optional static per-tensor scales”, and the same release also fixes “on-device state copy sizes for quantized KV caches” — if you have been using a quantized cache on an iGPU or a Metal backend, that is a bug fix worth taking. The full release breakdown is on the aifoss side: llama.cpp v1.0.0: what changed for self-hosters.
One caveat I will not paper over: q4_0 KV caches trade memory for numerical error in the attention path, and the exact quality impact depends on the model and the workload. This article does not claim a quality number for any of them.
What to actually buy
The card you need is decided by the context length you plan to use, not by the model name. Prices move daily, so check today’s price before you buy; the VRAM class is the part that does not change.
| Your situation | Card class | VRAM | Weights + KV at your context | Check today’s price |
|---|---|---|---|---|
| 30B-class model (Qwen3-30B-A3B Q4_K_M), full 40K context | 24 GB card | 24 GB | 19.2 GiB at 40,960 tokens (f16) | RTX 3090 24GB |
| Same, but you want to run two models or a bigger quant | 32 GB card | 32 GB | 19.2 GiB at 40,960 tokens, 13 GB spare | RTX 5090 32GB |
| 70B Q4_K_M at 8K context, f16 cache | 48 GB (two 24 GB cards) | 48 GB | 42.1 GiB at 8,192 tokens | RTX A6000 48GB |
70B Q4_K_M at 32K context, q8_0 cache | 48 GB (two 24 GB cards) | 48 GB | 44.6 GiB at 32,768 tokens | 2x RTX 3090 |
70B Q4_K_M at 128K context, q8_0 cache | 64 GB+ | 64 GB | 59.6 GiB at 131,072 tokens | RTX 6000 Ada 48GB |
| You are not sure yet what you will run | Rent it first | any | any | RunPod |
A note on the last row: renting an hour of a 48 GB GPU tells you more about your workload than any spec sheet, and it costs a few dollars rather than a few hundred. The VRAM tiers guide covers how to match a model to a card in the first place; this page is about the part that spec sheets leave out.
Honest take: what this arithmetic does not tell you
- It does not include the compute buffers. The KV cache and the weights are the two big line items, but the runtime also needs VRAM for activations, the CUDA context and scratch space. Assume a couple of gigabytes of overhead that is not in any table, including this one.
- It does not tell you the speed. A longer context makes attention more expensive, but the decode speed of a local model is dominated by memory bandwidth, not by how much context you keep. A 24 GB card does not become a slow card because you raised the context; it becomes a card that may not load.
- It does not cover the case where the context does not fit at all. Offloading the KV cache to NVMe is a separate technique with its own trade-offs, covered in the NVMe KV cache offloading guide.
- It does not apply to every runtime. The numbers above are the KV cache of a llama.cpp-style runtime. Other engines keep their own bookkeeping, and the fix for a load failure is still the same idea: shrink the context, shrink the cache type, or move up a card.
The order of operations when a model will not load is: check the weights against your VRAM first, then check the context length, then check the cache type. In that order, most “out of memory” errors on a 24 GB card are really “8K was fine, 32K was not”.
Sources
- Llama 3.3 70B Instruct Q4_K_M GGUF header —
llama.block_count80,llama.attention.head_count_kv8,llama.context_length131072,general.file_type15 — read from the file over an HTTP range request on November 19, 2026 - Llama 3.3 70B Instruct Q4_K_M file size — 42,520,398,816 bytes — Hugging Face tree API
- Qwen3-30B-A3B
config.json— 48 layers,num_key_value_heads4,max_position_embeddings40960 — Hugging Face - Qwen3-30B-A3B GGUF file sizes — Q4_K_M 18,556,686,912 bytes, Q5_K_M 21,725,581,888 bytes, Q8_0 32,483,932,736 bytes — Hugging Face tree API
- llama.cpp v1.0.0 release notes — FP8 KV caches (
-ctk/-ctv f8_e4m3), quantized KV cache state copy fix — GitHub releases API, published November 13, 2026 - GeForce RTX 3090 — 24 GB GDDR6X, 384-bit memory interface — NVIDIA product page
- GeForce RTX 4090 — 24 GB GDDR6X, 384-bit memory interface — NVIDIA product page
- GeForce RTX 5090 — 32 GB GDDR7, 512-bit memory interface — NVIDIA product page
Last updated November 19, 2026. Every architecture number above was read out of the GGUF headers or the Hugging Face API on November 19, 2026; every file size is the measured byte count from the Hugging Face tree API, not an estimate; the KV cache figures are arithmetic on those numbers and exclude runtime overhead. No model was run, no benchmark was performed, and no price is quoted because prices change daily — check the buyer’s guides before you buy. No paid API, no hosted service.
Was this article helpful?
Thanks for the feedback — it helps improve future articles.
Need hands-on help?
I offer 1-on-1 technical consulting for local AI setup, GPU selection, and AI coding tool configuration — same topics covered on this site.
Book a session — $49 / hour →Get the numbers before you buy
New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.