TL;DR: At the end of 2023 the best closed model beat the best open one by 17.5 MMLU points. By mid-2026 that gap on knowledge benchmarks is a rounding error, and the model that closes it — Qwen3.6 35B-A3B or Gemma 4 31B — fits on a single 24GB card you can buy used for around $1,000. The catch isn’t quality anymore; it’s electricity, VRAM ceilings, and the fact that the very top open models (Kimi K2.7, GLM 5.2) are still too big for one GPU.

	Run it locally (used RTX 3090)	Run it cloud (RunPod GPU)	Pay per token (API)
Upfront cost	~$1,000 for the card	$0	$0
Monthly cost	~$3–8 electricity	~$30–120 depending on hours	$0–100 depending on volume
Best for	Daily driver, privacy, always-on	Bursty heavy jobs, 70B+ models	Occasional use, frontier-only tasks
The catch	24GB VRAM ceiling; you own the failures	Meter never stops when idle	Your data leaves the building

Honest take: If you run a local model more than an hour a day, a used RTX 3090 pays for itself against cloud within a year and the open weights it runs are now good enough that you’ll rarely miss the closed frontier. Rent or use the API only for the handful of trillion-parameter models a single consumer card can’t hold.

The number that changed the argument

For two years the reflexive answer to “should I run a local model?” was “sure, but it won’t be as smart as GPT.” That’s no longer true in the way it used to be.

At the close of 2023, the strongest closed model scored roughly 88% on MMLU while the best open-weight alternative sat around 70.5% — a 17.5-percentage-point canyon (Let’s Data Science, SitePoint). By mid-2026 that canyon is effectively closed on knowledge benchmarks. On MMLU-Pro, the top open-weight models land within a few points of GPT-class and Claude-class systems, and on straight MMLU the difference between the best open and best closed model is now a fraction of a point rather than a chapter heading (BenchLM).

The important word is knowledge. The gap didn’t vanish everywhere at once — closed models still hold a lead on the hardest agentic and long-horizon reasoning tasks. But the “generalist assistant that knows things and writes well” job — the one most home-lab users actually want — is now a solved problem on hardware you already own or can buy for four figures.

For the why behind this — multi-token prediction, speculative decoding, and the MoE efficiency leap that made 35B-total models run like 3B ones — see why local LLMs got good in 2026. This article is about the what runs on what.

What “yesterday’s frontier” actually costs to run

The models that closed the gap split cleanly into three hardware tiers. The dividing line is VRAM, and the second-order line is memory bandwidth — because decode speed on a local LLM is bandwidth-bound, not compute-bound.

Tier	Model (example)	Quant / size	VRAM needed	Runs on	Real tok/s
Fits 16GB	Gemma 4 26B-A4B (QAT)	UD-Q4_K_XL, ~15GB	16GB	RTX 5060 Ti 16GB, 4060 Ti 16GB	~30–45
Fits 24GB	Qwen3.6 35B-A3B	Q4_K_M, ~21GB	24GB	Used RTX 3090, RTX 4090	~50–65 (up to ~100 on UD-Q4_K_XL)
Fits 24GB	Gemma 4 31B (QAT)	UD-Q4_K_XL, ~18GB	24GB	Used RTX 3090, RTX 4090	~25–40 (dense)
Multi-GPU / cloud only	Kimi K2.7, GLM 5.2, Qwen3-Coder 480B	2-bit+, 240GB–600GB	240GB+	4×+ GPUs or API	N/A on one card

The 24GB sweet spot: Qwen3.6 35B-A3B

This is the model that best demonstrates the closed gap. It’s a 35-billion-parameter mixture-of-experts model that activates only ~3B parameters per token, so at Q4_K_M it needs about 21GB of VRAM and decodes at roughly 50–65 tok/s on a used RTX 3090 — and higher on the Unsloth UD-Q4_K_XL dynamic quant, where community benchmarks report 80–101 tok/s on the same card (Hugging Face discussion, Will It Run AI).

For context, ~50 tok/s is roughly five to seven times faster than a person reads. The model is answering faster than you can consume it, on a card that costs less than a month of a heavy cloud-API habit. Our Qwen3.6 35B-A3B hardware guide has the full VRAM-vs-context breakdown.

The 16GB tier: Gemma 4 QAT

If you don’t have 24GB, Google’s quantization-aware-trained Gemma 4 checkpoints are the reason a 16GB card is still in the game. QAT cuts memory roughly 72% versus BF16 with near-original quality, so the 26B-A4B model now loads in about 15GB and the dense 31B in about 18GB (Unsloth docs, Google DeepMind). The 26B-A4B fits a 16GB laptop GPU; the 31B needs 24GB. One caveat worth repeating: use Unsloth’s pre-converted UD-Q4_K_XL GGUFs rather than hand-converting from Hugging Face — the naive q4_0 conversion reintroduces accuracy loss. The full memory map and the conversion trap are in our Gemma 4 QAT update.

The tier a single card can’t touch

The absolute top of the open-weight leaderboard — Kimi K2.7 (1T total / 32B active), GLM 5.2 (744B / 40B active), Qwen3-Coder 480B — did not get easier to run. The smallest usable quant of Kimi K2.7 is ~325GB, GLM 5.2’s 2-bit is ~241GB, and neither fits a single consumer GPU (see the Kimi K2.7 hardware guide). These are the models where the honest answer is “use the API or rent a multi-GPU pod.” The good news: you rarely need them. The 24GB-tier models closed the gap that matters for everyday use.

The real cost math: local vs cloud vs API

Here’s where the Tier-D honesty comes in. “Free” local inference isn’t free — it’s cheap, and cheap in a specific, quantifiable way.

A used RTX 3090 runs about $1,000 in mid-2026 — eBay used units averaged roughly $1,010–$1,050 with a range of $800 to $1,900 depending on condition and AIB model (Best Value GPU, XDA). Under sustained inference the card draws around 350W (Founders Edition TDP; factory-overclocked AIB models pull 380–420W) (NVIDIA forums). At $0.12/kWh, 350W is about $0.042 per hour of active generation.

Run the model two hours a day and that’s roughly $2.50/month in electricity plus idle draw — call it $3–8/month all-in. Amortize the $1,000 card over two years and you’re at about $45/month in year one, dropping to near-$5/month once it’s paid off.

Compare that to the alternatives:

Usage pattern	Local 3090	RunPod rental	Cloud API
Light (occasional)	$45/mo yr 1 (card amortized)	~$1–1.40/hr only when running	~$5–20/mo
Heavy daily (2–4 hr)	~$48/mo yr 1, ~$8/mo after	~$60–120/mo	~$50–100/mo
Frontier-only tasks	Can’t run 1T models	Best fit for 70B+ / MoE giants	Best fit — pay Opus 4.8 at $5/$25 per M

Cloud API pricing anchors the comparison: Claude Opus 4.8 is $5 per million input / $25 per million output tokens; GPT-5.5 runs higher at roughly $10/$40 (Claude pricing, Finout). At 10 million tokens a month that’s roughly $50–100 in API fees — every month, forever — versus a one-time card purchase that then costs pocket change to run. The deeper break-even analysis is in RunPod vs local GPU: rent or buy and the true cost of a 24/7 AI server.

The math flips on usage. Occasional user who fires up a model a few times a week: the API or a RunPod pod at ~$1/hr is cheaper and you skip the upfront cost — spin one up at RunPod and pay only for the minutes you use. Someone running a coding assistant, a RAG pipeline, or an always-on home server: the local card wins within a year and then keeps winning.

Which card for which budget

If you’re buying hardware specifically to exploit the closed gap, the decision is short:

• Best value, 24GB: a used RTX 3090 at ~$1,000. 936 GB/s bandwidth and 24GB is still the price-to-capability king for local AI — enough to run the 35B-A3B and 31B models that closed the gap. See why the used 3090 is still the value king.
• New, fastest 24GB+: the RTX 4090 (24GB) or RTX 5090 (32GB, 1,792 GB/s) if you want headroom for larger quants and faster prefill — at 3–4× the price.
• Budget 16GB: the RTX 5060 Ti 16GB at ~$430–480 runs the Gemma 4 26B-A4B QAT tier comfortably.

For the full per-card breakdown of which model and quant to target, see best local LLM for every RTX 50-series GPU. And if you’re picking the model rather than the card, our open-source LLM shootout ranks Qwen3.6, Gemma 4, Llama 4, GLM, and DeepSeek by VRAM tier.

The honest limits

Three things the “gap closed” headline glosses over:

1. Knowledge parity isn’t agentic parity. On multi-step tool use, long-context reasoning, and the hardest coding-agent benchmarks, closed frontier models still lead. If your workload is autonomous agents chaining dozens of tool calls, test before you commit — this is where the remaining gap lives. (For the coding-tool side of that comparison, see aicoderscope.com.)
2. The 24GB ceiling is real. The models that closed the gap are the 24GB-and-under ones. The trillion-parameter open models that top the leaderboard are cloud-scale hardware; a single consumer card doesn’t run them at any usable quant.
3. You own the failures. No SLA, no autoscaling, no one to page at 2 a.m. when a driver update breaks CUDA. Local means you’re the ops team. For a fully open, self-hosted stack walkthrough, aifoss.dev covers the FOSS tooling.

None of these undo the core result. For the everyday assistant, summarizer, coding copilot, and private-document Q&A that most home labs actually run, an open-weight model on a ~$1,000 card now delivers what cost $50–100/month in API fees a year ago — at the quality that used to require the closed frontier.

FAQ

Is the open-weight vs closed gap really zero now? On knowledge benchmarks like MMLU, effectively yes — the best open model is within a fraction of a point of the best closed one, down from 17.5 points in late 2023. On the hardest agentic and reasoning tasks, closed models still hold a single-digit lead. “Zero” applies to what most users do daily, not to every benchmark.

What’s the cheapest GPU that runs a gap-closing model? A 16GB card (RTX 5060 Ti 16GB, ~$430–480) runs Gemma 4 26B-A4B QAT at 15GB. A used RTX 3090 ($1,000, 24GB) runs the stronger Qwen3.6 35B-A3B at ~21GB and 50–65 tok/s.

How much does local inference actually cost per month? On a used RTX 3090 at 350W and $0.12/kWh, roughly $3–8/month in electricity for a couple hours of daily use. The card itself amortizes to about $45/month in year one, then near-$5/month.

Should I still pay for a cloud API? Yes, for two cases: occasional use (where you don’t want to buy hardware) and frontier-only tasks that need a trillion-parameter model or the last few points of agentic reasoning. For a daily-driver assistant, local wins on cost and privacy.

Can I run Kimi K2.7 or GLM 5.2 at home? Not on a single consumer GPU. The smallest usable quants are 240GB+. You need a multi-GPU rig, a high-RAM CPU-offload build, or the API. Stick to the 24GB-tier models for local.

Sources

1. Let’s Data Science — Open Source vs Closed LLMs: The 2026 Decision Framework — 2023 MMLU gap (17.5 pts) and 2026 convergence.
2. BenchLM — Best Open Source LLM in 2026 — MMLU-Pro open-vs-closed proximity.
3. SitePoint — Open-Source vs Commercial LLMs: The Complete Guide (2026) — benchmark convergence.
4. Hugging Face — Qwen3.6-35B-A3B discussion and Will It Run AI — Qwen 3.6 VRAM requirements — ~21GB Q4, 50–101 tok/s on RTX 3090.
5. Unsloth — Gemma 4 QAT docs and Google DeepMind — QAT for Gemma 4 — 26B ~15GB, 31B ~18GB, 72% memory cut.
6. Best Value GPU — RTX 3090 price tracker and XDA — Used RTX 3090 value king — ~$1,000 used, Jun 2026.
7. NVIDIA Developer Forums — RTX 3090 power draw — 350W TDP, 380–420W AIB under load.
8. Claude Platform — Pricing and Finout — Claude Opus 4.8 pricing 2026 — Opus 4.8 $5/$25 per M, GPT-5.5 ~$10/$40.

Prices and benchmarks verified as of July 2026. Hardware prices move weekly — check current listings before buying.

Recommended Gear

• RTX 3090 (used, 24GB) — best value for the 24GB gap-closing tier.
• RTX 4090 (24GB) — faster new option, same 24GB capacity.
• RTX 5090 (32GB) — headroom for larger quants and faster prefill.
• RTX 5060 Ti 16GB — budget entry that runs the Gemma 4 26B-A4B QAT tier.