KV Cache VRAM Cost: What 128K Context Actually Costs on a Consumer GPU (2026)

guidelocal-llmvramllama.cpp

TL;DR: Every token you keep in context costs you VRAM, on top of the weights, and the bill is bigger than most people expect. I read the headers of two real GGUF files today (November 19, 2026) over HTTP range requests and did the arithmetic from the model’s own numbers. For Llama 3.3 70B Instruct Q4_K_M (weights: 42,520,398,816 bytes = 39.60 GiB, measured), the KV cache costs 320 KiB per token at f16 — 2.50 GiB at 8K context, 10.00 GiB at 32K, and 40.00 GiB at the 131,072-token limit the model file itself declares. Switch the KV cache to q8_0 and those become 1.25 / 5.00 / 20.00 GiB; q4_0 quarters them again. That is the difference between a 48 GB card that runs 70B at 8K and one that cannot run 70B at 32K without quantizing the cache.

What you’ll be able to do:

  • Compute the exact KV cache footprint of any model from four numbers you can read out of the GGUF header.
  • Tell whether your “CUDA out of memory” error is the weights or the context window.
  • Pick a cache type (f16 / q8_0 / q4_0 / f8_e4m3) that fits the card you already own.
  • Size a card for the context length you actually need, instead of guessing.

Honest take: everything below is arithmetic on measured metadata, plus the published memory sizes from NVIDIA’s own product pages. I did not run a 70B model, I did not measure tok/s, and I make no quality claim about quantized KV caches. The formula is the well-known one (K and V per layer, per token), and every input to it is printed in the table so you can redo the sum for your own model.

The four numbers that decide everything

The KV cache is the part of a running model that holds the attention keys and values for every token in the current context. It grows linearly with the context length, and its size per token is fixed by the architecture:

bytes per token = 2 (keys + values)
                x layers
                x kv_heads
                x head_dim
                x bytes per element

Where head_dim = hidden_size / attention_heads, and bytes per element is 2 for f16, 1 for q8_0 (and for the new FP8 cache in llama.cpp v1.0.0), and 0.5 for q4_0.

All four architecture numbers are in the GGUF file. I read them straight out of the file header of bartowski/Llama-3.3-70B-Instruct-GGUF with a small HTTP range request — no download, no guesswork:

GGUF metadata keyValue
general.architecturellama
general.nameLlama 3.3 70B Instruct
general.file_type15 (MOSTLY_Q4_K_M)
llama.block_count80 layers
llama.embedding_length8192
llama.attention.head_count64
llama.attention.head_count_kv8
llama.context_length131072

So head_dim is 8192 / 64 = 128, and at f16:

2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes per token = 320 KiB per token

Multiply by the context length and you have the whole story. Note the 8 KV heads: Llama 3.3 70B uses grouped-query attention, so the KV cache is 8 heads wide while the query side has 64. That is why the number is manageable at all — a 64-head KV cache would be 8x bigger.

What that costs, at four context lengths

Same model, same weights (39.60 GiB), different context and cache type:

ContextKV at f16KV at q8_0KV at q4_0Weights + KV (f16)
8,192 tokens2.50 GiB1.25 GiB0.63 GiB42.10 GiB
32,768 tokens10.00 GiB5.00 GiB2.50 GiB49.60 GiB
65,536 tokens20.00 GiB10.00 GiB5.00 GiB59.60 GiB
131,072 tokens40.00 GiB20.00 GiB10.00 GiB79.60 GiB

Read the last row again. At the model’s own advertised 128K window, the KV cache at f16 is 40 GiB — as much VRAM as the entire card on a 48 GB machine, spent on keys and values that are mostly padding. The weights are not even the biggest line item anymore.

This is the mechanism behind the “my 24GB card ran the model fine until the context grew” experience: at 8K the KV cache is 2.5 GiB of a 24 GiB card, at 32K it is 10 GiB, and somewhere in between the model no longer loads.

The same math on a 30B MoE

Dense attention is not the only factor. A mixture-of-experts model with a small number of KV heads is much cheaper per token. From unsloth/Qwen3-30B-A3B’s config.json:

FieldValue
num_hidden_layers48
hidden_size2048
num_attention_heads32
num_key_value_heads4
max_position_embeddings40960
num_experts / num_experts_per_tok128 / 8

head_dim is 2048 / 32 = 64, so at f16:

2 x 48 x 4 x 64 x 2 bytes = 49,152 bytes per token = 48 KiB per token

And the weights, measured from the Hugging Face tree API for unsloth/Qwen3-30B-A3B-GGUF:

QuantizationFile size (measured)GiB+ KV at 40,960 tokens (f16)
Q4_K_M18,556,686,912 bytes17.28 GiB19.16 GiB
Q5_K_M21,725,581,888 bytes20.23 GiB22.11 GiB
Q8_032,483,932,736 bytes30.25 GiB32.13 GiB

A 24 GB card runs Qwen3-30B-A3B at its full 40,960-token context in Q4_K_M with about 5 GiB to spare — and that spare headroom is what the compute buffers, the CUDA context and the runtime need. The 70B model on the same card is not a context problem at all; it does not fit at any context length, because the weights alone are 39.60 GiB.

Choosing the cache type

llama.cpp exposes the cache type per tensor:

$ llama-server -m model.gguf -c 32768 --cache-type-k q8_0 --cache-type-v q8_0

The cache types you will actually see in the wild:

Cache typeBytes per elementKV at 32,768 tokens (70B)When to use it
f16 (default)210.00 GiBYou have the VRAM and want the reference behaviour
q8_015.00 GiBThe default choice for a long context on a card that is already tight
q4_00.52.50 GiBYou are squeezing the last gigabyte; more quantization error in the attention path
f8_e4m315.00 GiBNew in llama.cpp v1.0.0 (released November 13, 2026): OCP-format FP8 cache with optional static per-tensor scales

The v1.0.0 release notes list the new FP8 KV cache under core changes as “FP8 KV caches (-ctk/-ctv f8_e4m3) with optional static per-tensor scales”, and the same release also fixes “on-device state copy sizes for quantized KV caches” — if you have been using a quantized cache on an iGPU or a Metal backend, that is a bug fix worth taking. The full release breakdown is on the aifoss side: llama.cpp v1.0.0: what changed for self-hosters.

One caveat I will not paper over: q4_0 KV caches trade memory for numerical error in the attention path, and the exact quality impact depends on the model and the workload. This article does not claim a quality number for any of them.

What to actually buy

The card you need is decided by the context length you plan to use, not by the model name. Prices move daily, so check today’s price before you buy; the VRAM class is the part that does not change.

Your situationCard classVRAMWeights + KV at your contextCheck today’s price
30B-class model (Qwen3-30B-A3B Q4_K_M), full 40K context24 GB card24 GB19.2 GiB at 40,960 tokens (f16)RTX 3090 24GB
Same, but you want to run two models or a bigger quant32 GB card32 GB19.2 GiB at 40,960 tokens, 13 GB spareRTX 5090 32GB
70B Q4_K_M at 8K context, f16 cache48 GB (two 24 GB cards)48 GB42.1 GiB at 8,192 tokensRTX A6000 48GB
70B Q4_K_M at 32K context, q8_0 cache48 GB (two 24 GB cards)48 GB44.6 GiB at 32,768 tokens2x RTX 3090
70B Q4_K_M at 128K context, q8_0 cache64 GB+64 GB59.6 GiB at 131,072 tokensRTX 6000 Ada 48GB
You are not sure yet what you will runRent it firstanyanyRunPod

A note on the last row: renting an hour of a 48 GB GPU tells you more about your workload than any spec sheet, and it costs a few dollars rather than a few hundred. The VRAM tiers guide covers how to match a model to a card in the first place; this page is about the part that spec sheets leave out.

Honest take: what this arithmetic does not tell you

  • It does not include the compute buffers. The KV cache and the weights are the two big line items, but the runtime also needs VRAM for activations, the CUDA context and scratch space. Assume a couple of gigabytes of overhead that is not in any table, including this one.
  • It does not tell you the speed. A longer context makes attention more expensive, but the decode speed of a local model is dominated by memory bandwidth, not by how much context you keep. A 24 GB card does not become a slow card because you raised the context; it becomes a card that may not load.
  • It does not cover the case where the context does not fit at all. Offloading the KV cache to NVMe is a separate technique with its own trade-offs, covered in the NVMe KV cache offloading guide.
  • It does not apply to every runtime. The numbers above are the KV cache of a llama.cpp-style runtime. Other engines keep their own bookkeeping, and the fix for a load failure is still the same idea: shrink the context, shrink the cache type, or move up a card.

The order of operations when a model will not load is: check the weights against your VRAM first, then check the context length, then check the cache type. In that order, most “out of memory” errors on a 24 GB card are really “8K was fine, 32K was not”.

Sources

Last updated November 19, 2026. Every architecture number above was read out of the GGUF headers or the Hugging Face API on November 19, 2026; every file size is the measured byte count from the Hugging Face tree API, not an estimate; the KV cache figures are arithmetic on those numbers and exclude runtime overhead. No model was run, no benchmark was performed, and no price is quoted because prices change daily — check the buyer’s guides before you buy. No paid API, no hosted service.

Was this article helpful?

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.