Aug 13, 2026

Best Local LLMs for 32GB VRAM in 2026: The First Single Card That Holds a 70B

By RunAIHome Team · 15 min read

32gb-vramlocal-llmrtx-5090radeon-ai-pro-r9700qwen3llama-3-3-70bgemma-4mac-mini-m4-proollamabuying-guide

TL;DR: 32GB is where the 24GB tier’s compromises disappear — Qwen3.6-35B-A3B runs at Q5 with real context instead of Q4 on a knife’s edge — and where a dense 70B finally sits fully resident on one card, at IQ3 quantization and with honest caveats. The surprise is the hardware: the $1,299-MSRP Radeon AI PRO R9700 made this tier affordable while the RTX 5090 drifted past $4,300.

	Qwen3.6-35B-A3B (Q5_K_M)	Qwen3.6-27B (Q6_K)	Llama 3.3 70B (IQ3_XXS)
Best for	Speed and agents — ~194 tok/s on a 5090	Coding — 77.2% SWE-bench, now at a better quant	The 70B experiment 24GB owners can’t run
File size	25.9GB	22.5GB	27.7GB
Context room on 32GB	16K–32K with KV quantization	32K+ without tricks	~8K–16K, KV quantization required
The catch	Q6_K (30.9GB) does NOT fit — stop at Q5	Dense speed: 60–90 tok/s on a 5090, less elsewhere	IQ3 quality; a 27B–32B at Q4+ often beats it

Honest take: buy this tier for comfort on the 27B–35B class, not for the 70B — pull Qwen3.6-27B at Q6_K first, add the 35B-A3B at Q5 for agent work, and treat the 70B as a weekend experiment. And unless you also game at 4K, the card to buy is the R9700 at MSRP, not a $4,400 RTX 5090.

The 32GB tier is the strangest rung on the VRAM ladder. It contains exactly one consumer GeForce card — the RTX 5090 — plus a workstation card most buyers overlook (AMD’s Radeon AI PRO R9700) and, loosely, the 48GB unified-memory Macs whose GPU-usable slice lands in the same 30s. The full ladder is in our VRAM tier hub, the 24GB edition covers the rung below (start there if you’re choosing between them), the 16GB, 12GB, 8GB, and 6GB editions complete the series, and the VRAM calculator will check any model + context combination against your exact card before you commit to a 28GB download.

Here’s the thing our own 24GB guide told you yesterday, and it’s still true: 8 extra GB does not unlock a new model class the way 12GB→16GB→24GB each did. A dense 70B at Q4_K_M is ~43GB — it fits a 5090 no better than a 3090. What 32GB buys is different, and whether it’s worth the money depends entirely on which card you buy it in.

The real VRAM budget on a “32GB” card

Same arithmetic as every tier. The OS, compositor, and browser skim a few hundred MB; budget 31 to 31.5GB usable. Weights (roughly the GGUF file size) plus KV cache (grows linearly with context) plus a few hundred MB of compute buffers must fit inside that.

The tier splits into three bands. Files up to ~23GB (Qwen3.6-27B Q6_K, Gemma 4 31B QAT, everything from the 24GB tier) leave 8GB+ of cache room — long-context territory without any tricks. The 25–28GB band (35B-A3B at Q5, 70B at IQ3_XXS, Qwen3.6-27B at Q8_0) works, but you’re back to KV-cache discipline. And the 29–31GB band — Qwen3.6-35B-A3B Q6_K at 30.9GB is the classic — is the trap: it loads, answers a short prompt, and falls over on the first long one. More on that below.

What 32GB actually buys over 24GB

Three things, in descending order of how often you’ll feel them.

First: the 24GB tier’s signature trap disappears. On a 3090, Qwen3.6-35B-A3B’s 21GB Q4_K_M file leaves under 2GB for cache — 16K context barely fits and 32K doesn’t, which is why the 24GB guide spends a whole section on KV-quantization triage. On 32GB the same file leaves ~10GB: that’s 32K context with an unquantized cache. The model you already run stops needing surgery.

Second: a quant level, or two. Qwen3.6-27B jumps from Q4_K_M (16.8GB) to Q6_K at 22.5GB, or even Q8_0 at 28.6GB — and the 35B-A3B steps up from Q4 to Q5_K_M at 25.9GB. Our quantization quality data says Q4→Q6 is a real, if subtle, improvement on reasoning-heavy work; Q6→Q8 is mostly placebo. Q6_K on the 27B with 8GB of cache room is arguably this tier’s single best configuration.

Third: the 70B door cracks open. This is the headline nobody should buy the tier for, so let’s do it honestly.

The 70B on one card: real, and overrated

A dense 70B at Q4_K_M is ~43GB. At Q3_K_S it’s 30.9GB — a nominal fit with ~0.5GB to spare, which is no fit at all. The quant that actually works is IQ3_XXS at 27.7GB: fully resident with ~3.5GB left for cache, good for roughly 8K–16K context with a q8_0 cache.

Speed is genuinely fine. Fully-resident decode reports on the RTX 5090 run from the 40s of tok/s up to claims near the card’s ~65 tok/s bandwidth ceiling for a file this size (Markaicode’s LM Studio run reports ~70, which we’d treat as best-case; ModelFit’s 5090 data frames the card as “runs 32B Q4, 70B Q3”). Plan around 40 and you won’t be disappointed. Compare that with the 8–14 tok/s partial-offload slog that is 70B life on a 24GB card, and this tier is the first time a 70B feels like a daily-usable model on one consumer-class card.

Now the honest part. IQ3_XXS is an aggressive quant — you’re keeping the 70B’s breadth of knowledge but sanding off precision, and on math- and code-heavy tasks the community’s long-standing rule holds: a 27B–32B at Q4 or better usually beats a 70B at IQ3. Qwen3.6-27B scores 77.2% on SWE-bench Verified at quants this tier runs comfortably; Llama 3.3 70B at IQ3_XXS won’t out-code it. Where the 70B earns its slot is breadth — general knowledge, nuanced writing, instruction-following on messy prompts. Run the experiment; most people come back to the 27B.

One practical warning: Ollama’s default llama3.3:70b tag ships a Q4-class file that will not fit — it’ll silently spill and crawl. For the IQ3 experiment, pull the bartowski IQ3_XXS GGUF through llama.cpp or LM Studio instead.

The picks, by what you actually do

The default: Qwen3.6-27B at Q6_K

Everything the 24GB guide said about the dense 27B — best open coding model on one card, 77.2% SWE-bench Verified — plus the quant bump only this tier affords: Q6_K at 22.5GB with ~9GB of cache room, which is 32K+ context and zero KV-cache ceremony. On an RTX 5090 it decodes at 60–90 tok/s at Q4-class quants — the dense-model sluggishness that defines it on a 3090 (~40 tok/s) mostly evaporates at 1,792 GB/s. On the R9700’s 640 GB/s, expect roughly 3090-class pace at a Q6 file: usable, not thrilling.

The speed pick: Qwen3.6-35B-A3B at Q5_K_M

The 24GB tier runs this MoE at Q4 with a starved cache; 32GB runs it at Q5_K_M (25.9GB) with ~5GB to spare. Only ~3B parameters activate per token, so it decodes like a small model: llama-bench measured 194 tok/s on an RTX 5090 on the 35B-A3B class, and AMD’s R9700 does 127–156 tok/s on the same class — MoE decode is where the cheap card punches hardest. Full VRAM math and tuning flags are in the 35B-A3B guide.

Do not pull Q6_K. It’s a 30.9GB file against a 31.5GB budget — see the trap section.

The general model: Gemma 4 31B QAT with actual context

Google’s 31B QAT build (~18GB per Unsloth’s table, 19GB Ollama tag) was the 24GB tier’s multimodal flagship; here it becomes something better — an 18GB model with 13GB of cache room, which means huge-context document work a 3090 can’t offer it. Same rule as always (QAT guide): use the released QAT files, don’t convert your own.

The two-model trick

31.5GB fits Qwen3.6-27B Q4 (16.8GB) and a 9B-class model (~6GB) resident simultaneously, with cache room for both. A coding model and a fast summarizer/draft model that never swap — or an LLM plus an image model: MiniMax H3’s Q5-class video GGUF is 23.9GB, and its NVFP4 build peaked at ~26.9GB in measured community runs on an RTX 5090, both squarely 32GB-tier footprints. FLUX.2 at BF16 (~35GB) still doesn’t fit, for the record.

What you cannot run on 32GB (stop trying)

Model	Footprint	What it actually needs
Llama 3.3 70B at Q4_K_M	~43GB	48GB — dual 3090s or dual R9700s
Llama 4 Scout	~67GB	Multi-GPU or unified memory
GPT-OSS 120B	72.7GB	Rented cloud
Qwen3.6-35B-A3B Q6_K	30.9GB	Technically loads; dies on the first long prompt
GLM 5.2 / Kimi K3	100GB–1.56TB	Not your house

The pattern to internalize: 32GB upgraded your quants and context, not your model class. The next class boundary is 48GB, and no single consumer card gets there in 2026.

The trap at this tier: the 29–31GB file

Every tier has one. At 24GB it was the 21GB MoE; at 32GB it’s any file in the 29–31GB band — Qwen3.6-35B-A3B Q6_K (30.9GB), Qwen3.6-27B Q8_0 (28.6GB) at long context, 70B Q3_K_S (30.9GB). They load. They answer a short prompt. Then a 16K-token agent session adds 2–3GB of KV cache, and you either hit CUDA out of memory or — worse, on Windows — the driver silently pages into system RAM and generation collapses to single digits, the sysmem-fallback failure that ollama ps can’t see.

The fix order, same as every tier, bigger numbers:

Buy quant headroom, not quant maximum. Cap weights at ~26GB (Q5 on the 35B, Q6 on the 27B, IQ3_XXS on the 70B). The quality data says the last quant level is worth far less than the context it costs.
Quantize the KV cache when you do run the 25–28GB band: OLLAMA_FLASH_ATTENTION=1 + OLLAMA_KV_CACHE_TYPE=q8_0, set on the service — details in the tuning guide.
Check ollama ps after the first long prompt. 100% GPU or it doesn’t count.

Which 32GB machine in August 2026

This is where the tier gets interesting, because the obvious card is the wrong one for most buyers.

Machine	Price (Aug 2026)	Bandwidth	Verdict
Radeon AI PRO R9700	$1,299 MSRP; resale $1,522–$2,139	640 GB/s	The value story of the tier — if you can find MSRP stock
RTX 5090	cheapest new listing $4,399; ~$4,329 on Amazon; used ~$3,999	1,792 GB/s	The speed ceiling, priced at 2.5× everything else here
Mac Mini M4 Pro 48GB	~$1,799 all-in	273 GB/s	~36GB GPU-usable, silent, 40W — and slow on dense models
Mac Studio M4 Max 36GB	~$2,999 base	546 GB/s	The in-between: 3090-class bandwidth, unified-memory flexibility

The RTX 5090’s price history is the market’s problem, not yours to fix: BuySellRam’s August report puts the cheapest retail listing at $4,399 against a $1,999 launch MSRP, with used cards within 14% of new — there is no used discount to hide in. It is a magnificent card (our 5090 vs 4090 breakdown) and at these prices it makes sense for exactly two groups: people whose workload is prefill-heavy (huge-context agents, batch pipelines, image/video generation) where its compute lead is architectural, and people who also want the fastest gaming card on Earth.

For everyone else, the R9700 at $1,299 MSRP is the same 32GB for less than a third of the money — our full R9700 review has the fine print. The honest trade: MoE decode is superb (127–156 tok/s on the 35B class), dense decode is 3090-class, but prefill is 2.6–3.4× slower than NVIDIA — long-prompt agent work will feel it — and Ollama on RDNA4 still wants a Vulkan workaround, though llama.cpp and ROCm 7.2 are solid. Watch stock: resale listings run $1,522–$2,139, and above ~$1,700 the value argument thins fast.

The Mac path is a different philosophy. A 48GB Mac Mini M4 Pro gives its GPU roughly 75% of unified memory — ~36GB, this tier’s capacity — at $1,799 in a silent 40W box, but 273 GB/s means dense 27B–32B models run at 15–28 tok/s, not 60–90. The M4 Max Mac Studio doubles that bandwidth for a grand more. Macs win on capacity-per-dollar-per-watt and lose on speed; if your models are MoE, the gap narrows.

If you want to run the next tier up

The next real boundary is 48GB — the full-quality 70B tier (Q4_K_M at ~43GB, fully resident). No single consumer card gets there in 2026; the routes are:

Two used RTX 3090s (~$2,496 at ~$1,248 each) — the classic path; read the dual-3090 PCIe guide before buying a motherboard that can’t feed them.
Two R9700s ($2,598 at MSRP) — 64GB of new, warrantied VRAM for less than one 5090; the cheapest new-hardware path to full-GPU 70B Q4 right now.
Rent it: a 48GB A6000-class card on RunPod runs well under a dollar an hour, which for occasional 70B work beats owning for years.

The GPU buying guide has the full ladder logic, including why stepping down to a used 3090 at ~$1,248 is still the right call for most people reading this page.

Setup that works, start to finish

Tested against Ollama v0.32.6 (August 4, 2026):

# the default: 27B at the quant this tier earns (pull the exact GGUF from HF)
ollama pull hf.co/unsloth/Qwen3.6-27B-GGUF:Q6_K

# the speed pick — Q5, NOT Q6 (30.9GB does not fit)
ollama pull hf.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF:Q5_K_M

# the general/multimodal pick (Ollama's own QAT tag)
ollama pull gemma4:31b-it-qat

# confirm residency after a LONG prompt, not a short one
ollama ps
NAME                                          ID          SIZE     PROCESSOR    UNTIL
hf.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF...  a3f81c22    29 GB    100% GPU     4 minutes from now

Run the 27B with --num-ctx 32768 — the 2048-token default wastes the exact headroom you paid for. The 70B experiment goes through llama.cpp or LM Studio with the bartowski IQ3_XXS file, not Ollama’s default tag. For a coding-agent backend on this hardware, aicoderscope.com covers the Cline/Continue.dev side; aifoss.dev tracks the open-source serving stack.

FAQ

Is the RTX 5090 worth it over a 24GB card? At $4,399+ in August 2026, only if you need its prefill/compute lead (agents with huge prompts, image/video work) or its gaming. For pure LLM decode, a used $1,248 RTX 3090 runs most of the same models at Q4, and the R9700 runs all of them for $1,299. The 24GB guide called the 5090 “priced for someone else”; nothing this week changed that.

Does a 70B beat Qwen3.6-27B at this tier? For coding and math, usually no — IQ3_XXS costs the 70B too much precision, and the 27B’s 77.2% SWE-bench at Q6 stands. For broad-knowledge chat and writing, the 70B has a real edge. Run both for a week; keep the one you stop thinking about.

What’s the catch with the R9700? Prefill speed (2.6–3.4× slower than NVIDIA — long prompts drag), an Ollama workaround on RDNA4, and MSRP stock that comes and goes. If your stack is llama.cpp/LM Studio and your models are MoE, the catch is small. Full detail in the R9700 review.

Mac Mini M4 Pro 48GB or a 32GB GPU? The Mac if you want silence, 40W, and capacity for the occasional big model at patient speeds; the GPU if you want 60–194 tok/s on the models you run daily. The Mac-vs-5090 head-to-head has the full numbers.

What about image and video generation on 32GB? The comfortable tier for it: FLUX.2 at FP8/GGUF without offload, MiniMax H3 at Q5-class GGUF (23.9GB) or NVFP4 — the NVFP4 speed path is Blackwell-only, one real RTX 5090 exclusive. BF16 FLUX.2 (~35GB) is the one that still doesn’t fit.

Recommended Gear

Radeon AI PRO R9700 — 32GB at $1,299 MSRP; the tier’s value pick for llama.cpp/MoE stacks
RTX 5090 — 1,792 GB/s; the speed ceiling if the price doesn’t scare you (it should)
RTX 3090 used — two of them = the 48GB tier for the price of half a 5090
Mac Mini M4 Pro 48GB — the silent 40W route to ~36GB of model room
Mac Studio M4 Max — 546 GB/s unified memory when you want Mac and speed

Sources

Last updated August 13, 2026. Prices and specs change; verify current rates before purchasing. Tokens/sec figures vary with runtime, quantization, context length, and tuning; treat the low end of each range as the planning number. The 70B-on-5090 decode figure and R9700 dense-model pace are reported/bandwidth-scaled estimates, not our own benchmarks.

Was this article helpful?