Best Open-Source LLMs for Your Home Lab, October 2026: The Month Models Started Shipping at 2-Bit
TL;DR: Nine open-weight releases landed in the four weeks since the September edition, and the pattern flipped: vendors now ship 2-bit quants as the official artifact, not a community afterthought. Nemotron 3.5 Bolt puts a 74B agent model on a 24GB card in a 22.02GB official GGUF; Mellum 2.1 and MiMo’s 9B distill give the 12GB tier its first real movement since June; and the new open-weights intelligence leader is MIT-licensed — but weighs 1.02 trillion parameters.
| Nemotron 3.5 Bolt 74B | Mellum 2.1 | MiMo-V2.6-Pro | |
|---|---|---|---|
| Best for | 262K-context agent runs on one 24GB card | Agentic coding on a 12GB card | Frontier work via API |
| License | OpenMDW-1.1 | Apache 2.0 | MIT |
| Smallest usable quant | 22.02GB IQ2_XXS (the official release format) | 7.52GB Q4_K_M | ~350GB-class — 1.02T total |
| Runs on your hardware? | Yes — fits a used RTX 3090 with KV tricks | Yes — fits a used RTX 3060 | No |
| The catch | Scores a hair below the 35B MoE you already run | All benchmarks vendor-reported so far | MIT license, datacenter physics |
Honest take: On a 24GB card, Qwen3.8-27B with the MTP flag keeps the crown for a second month — pull Nemotron Bolt alongside it only if your work lives past 128K context. The month’s real movement is at 12GB, where two coding models finally give a $280 used card a serious job.
September’s edition closed on a sour trend: four of seven releases carried custom licenses, and the newest shipped no weights at all. October answered with the opposite problem. Weights are flowing again — MIT at the very top of the leaderboard, day-one GGUFs, same-day llama.cpp merges — but almost everything worth bragging about needs more memory than any machine this site has ever recommended. Here’s the four-week accounting: what shipped, what it runs on, which tiers actually moved, and the September claims that didn’t survive the month.
Four weeks, nine releases, and a new official format: 2-bit
Everything that hit the scoreboard between September 21 and October 17, with the columns that decide whether you can run it:
| Model (weights date) | Total / active | License | Smallest usable quant | What actually runs it |
|---|---|---|---|---|
| MiMo-V2.6-Pro (Sep 21) | 1.02T / 42B | MIT | ~350GB-class | Datacenter node |
| MiMo-V2.6-Flash (Sep 21) | 309B / 15B | MIT | 165.5GB native ~4-bit | 256GB-class machine |
| MiMo-V2.6-Distill-Qwen-9B (Sep 21) | 9B dense | MIT | 5.84GB Q4_K_M | Any 8GB+ card |
| Kolibri-1 (Oct 3) | 78.1B / 3.46B | Apache 2.0 | 47.45GB Q4_K_M | 64GB RAM box, 64GB Mac |
| Mellum 2.1 (Oct 8) | 12B / 2.5B | Apache 2.0 | 7.52GB Q4_K_M | One 12GB card |
| Nemotron 3.5 Bolt (Oct 15) | 74B / 9B | OpenMDW-1.1 | 22.02GB IQ2_XXS — official | One 24GB card |
| Nemotron 3.5 Super VL (Oct 15) | 120B / 12B | OpenMDW-1.1 | 86.9GB Q4 | 96–128GB box |
| Muse Spark Open (Oct 15) | 816B / 42B | Muse Model License | ~256GB 2-bit floor | 2× M5 Ultra 512GB — not orderable |
| Step 5 Preview (Oct 17 confirmed) | 600B / 27B | StepFun License 1.0 | ~217GB community Q2 | 4× DGX Spark, measured |
Two more were announced but haven’t shipped weights: Mistral Large 4 (1.05T/49B, promised by end of October) and Reflection Beam (501B/23B, Apache 2.0 promised). Both get a row in next month’s table or a correction, whichever they earn.
The quant column is the story. NVIDIA didn’t just tolerate a 2-bit Bolt — it published the IQ2_XXS GGUF as the release artifact and ran its benchmark table at 2-bit. Meta shipped official Q2_K GGUFs for Muse Spark on day one. Step 5’s only measured local run is a 2-bit-class EXL3 hybrid. And llama.cpp spent October 17 shipping faster CUDA MMVQ kernels for exactly these low-bit types plus Hexagon support for Q1/Q2. A year ago 2-bit was a desperation move; this month it’s the format frontier MoEs are designed to be served in. The practical consequence: when you read a VRAM requirement now, ask which quant it assumes — “fits 24GB” increasingly means “at the 2-bit the vendor benchmarked,” not the Q4_K_M you’d pull by habit.
The 24GB tier: the champion holds, but gets a long-context stablemate
September’s verdict — Qwen3.8-27B Q4_K_M (16.8GB) plus the MTP flag, 41→66 tok/s on a used RTX 3090 — survives October intact. Nothing released this month beats it as a daily driver, and one thing we warned about turned out to be a false alarm (next section).
What changed is the specialist slot. Nemotron 3.5 Bolt is a 74B/9B-active hybrid Mamba-MoE whose official 22.02GB IQ2_XXS loads on a 24GB card with room for about 128K of context at FP16 KV — push toward its tested 262K and you’ll want -ctk q8_0 -ctv q8_0 to stay inside the card. NVIDIA’s own model card is unusually honest about where it stands: Bolt at 2-bit scores 87.82 on PinchBench against 88.07 for Qwen3.6-35B-A3B at BF16 — the MoE most 24GB owners already run, at half the download. The number that justifies the pull is long context: 86.43 on AA-LCR versus the 35B’s 61.06 (vendor-reported, both). With ~9B active parameters per token, our bandwidth-ceiling estimate puts a 3090 at roughly 70–80 tok/s — estimated, no community bench yet — against the 107 tok/s the 35B measures. The full Bolt guide has the KV math and the one genuine 32GB carrot: Bolt’s DFlash draft model peaks at 26.4GiB, so the speculative-decoding speedup is physically reserved for RTX 5090-class cards — the first model-shaped argument for the 32GB tier we’ve seen.
So the 24GB loadout as of October 18 is three models, not one: Qwen3.8-27B for quality, Qwen3.6-35B-A3B for speed, Bolt for anything past 128K. The 24GB tier guide walks the trade-offs.
The Ollama MTP scare didn’t reproduce
Our Ollama 0.34 speculative-decoding coverage relayed a September bug report (ollama #18541) claiming MTP drafting collapsed qwen3.8:27b from ~75 to ~17 tok/s, with draft_num_predict 0 as the workaround. That issue is now closed, and the resolution is worth more than the scare: a repo collaborator benchmarked the identical model across five Ollama versions (0.33.2 through 0.34.2) and measured 107–115 tok/s with 0.939 draft acceptance on every single one — the regression never reproduced. If you applied the workaround, undo it; you’re leaving roughly a third of your decode speed off. Thirty seconds verifies your own install:
ollama run qwen3.8:27b --verbose "Explain KV cache quantization in two sentences."
# healthy MTP on a 24GB Ampere/Ada card ends with something like:
# eval rate: 107.43 tokens/sIf your eval rate sits near the old ~41–48 tok/s baseline instead, your local copy predates the MTP tensors — ollama pull qwen3.8:27b to re-fetch, then rerun the check.
The 12GB tier finally moved — twice in three weeks
Every edition since June has ended the small-VRAM section with “nothing new targets this tier.” October broke the streak from two directions.
JetBrains Mellum 2.1 (Oct 8, Apache 2.0) is a 12B/2.5B-active MoE whose Q4_K_M lands at 7.52GB, with a sliding-window attention layout that keeps even 131K context around 1.8GiB of KV — comfortable on 12GB. JetBrains reports 47% on SWE-bench Verified, up from the original Mellum’s ~2%.
Xiaomi’s MiMo-V2.6-Distill-Qwen-9B (Sep 21, MIT) attacks from below: a Qwen3.5-9B fine-tune distilled from the 1T-parameter Pro, 5.84GB at Q4_K_M in bartowski’s GGUFs, reporting 61.1 on SWE-bench Verified and 44.6 on SWE-bench Pro against its base model’s 60.0 and 32.0 — plus a cybersecurity-bench jump from 5.7 to 31.3. Since it keeps the stock Qwen3.5 architecture, it loads in any llama.cpp, Ollama, or LM Studio build from months ago — no merge to wait for.
Both score sheets are vendor-reported, and this is exactly the situation where harness differences bite: JetBrains’ own published table scores Qwen3.5-9B differently than Qwen’s card does, so a Mellum-47 vs MiMo-61.1 comparison across two vendors’ harnesses settles nothing. What the two releases establish jointly is weaker but real: serious labs are now shipping agentic coding models sized for 8–12GB cards, which makes a used RTX 3060 12GB at $260–$296 a working entry ticket rather than a compromise. Run your context plans through the VRAM calculator first — at 12GB, KV cache is the budget, not the weights. The 12GB tier guide has the full picks; for wiring either model into Cursor or Cline as a local backend, our sister site aicoderscope.com covers the BYOK setup.
Mainline llama.cpp caught up — two September complaints resolved
GLM-5.3-Flash is in. September’s “what didn’t happen” section tracked the three-way PR standoff that left Unsloth’s branch as the only consumer path to the 93GB-class MoE. On October 1, maintainers closed the lead PR with “support is already in” — the architecture landed upstream through other work. Caveats survive: GGUFs converted under the early glm5next arch name need re-converted or updated files (Unsloth’s docs now demo a 3-bit UD-IQ3_XXS sized for 128GB devices), and there’s still no official Ollama or LM Studio listing — ollama run hf.co/... community pulls are the path. Progress, not polish; details in our GLM-5.3-Flash guide.
Kolibri-1 loads without patches now. When we covered Aleph Alpha’s 78B MoE on October 10, the subtitle was “the MoE your tools can’t load yet” — mainline llama.cpp threw unknown model architecture: 'kolibri1' and the workaround was a third-party patch. Fourteen days after the weights dropped, PR #30673 merged (build b11709, October 17), with the full exotic layout — 384 routed experts, sigmoid routing, alternating 513-token sliding-window and NoPE full-attention layers — upstreamed and covered by arch tests. The hardware verdict from our guide doesn’t change (47.45GB at Q4_K_M still means a 64GB-RAM box or big-memory Mac, and Qwen3.6-35B-A3B still beats it on English coding at half the size), but the asterisk is gone: stock llama.cpp runs it, and Ollama/LM Studio support inherits on their next llama.cpp bump. The measured numbers we printed — 11.9–14.9 tok/s CPU-only on a 7800X3D — came from the patched fork; expect mainline benches to land within weeks.
Add Bolt and Muse Spark’s same-day merges and the pattern is clear: day-one (or at worst week-two) mainline support has become table stakes for a serious open-weights launch. The GLM-5.3-Flash saga — three and a half weeks in branch purgatory — already reads like the old days.
Tier by tier: what to run in October 2026
| Your memory | Run this | Expect | Tier guide |
|---|---|---|---|
| 8–12GB | Mellum 2.1 (7.5GB) or MiMo 9B distill (5.8GB) for code — new; Gemma 4 12B QAT general | First credible agent coders at this tier | 12GB guide |
| 16GB | Gemma 4 26B-A4B QAT (~15GB); Codestral 2 (13.3GB) | Unchanged since September | 16GB guide |
| 24GB | Qwen3.8-27B + MTP (16.8GB); Bolt IQ2_XXS (22GB) past 128K ctx — new | 41–66 tok/s; Bolt ~70–80 est. | 24GB guide |
| 32GB (RTX 5090) | Above plus Bolt + DFlash drafter (26.4GiB) — new | First model-shaped 32GB reason | 32GB guide |
| 48–64GB | Llama 3.3 70B Q4 resident; Kolibri-1 now mainline on 64GB | 7–10 tok/s (70B); ~12–15 CPU (Kolibri) | 48GB guide |
| 96GB | Inkling-Small 2-bit (87.9GB); Super VL 120B Q4 (86.9GB) — new | Two frontier-adjacent fits now | 96GB guide |
| 128GB unified | Qwen3.8-Flash-Next; GLM-5.3-Flash 3-bit now mainline; DeepSeek V4 Flash via DS4 (96.5GB) | 22–82 tok/s; DS4 39.4 on M5 Max | 128GB guide |
| 256GB+ | Muse Spark Q2 (256GB floor); Step 5 Q2 (~217GB); MiMo Flash (165GB) | 18–30 tok/s measured, config-dependent | Muse Spark guide |
And if your card tops out at 4GB of VRAM, October still delivered: VibeThinker-3B is a 1.93GB MIT-licensed download whose vendor-claimed AIME score embarrasses models 200× its size — budget 60K+ thinking tokens per answer, which is a patience floor, not a hardware one.
What didn’t happen (October’s corrections ledger)
The frontier crown changed hands — toward MIT, away from your hardware. Xiaomi’s MiMo-V2.6-Pro posted a 46 on the Artificial Analysis Intelligence Index — above GLM-5.3 (45), Step 5 Preview and Kimi K3 (44), and well clear of Muse Spark Open (39) — under a clean MIT license with no revenue triggers. September’s “less open at the top” trend inverted in one release. The hardware story didn’t: 1.02T total parameters is a datacenter problem whatever the license says. Third month running, the conclusion survives: rent the frontier through an API, let your silicon run the sub-100B class it’s good at. The self-hosting stacks for the big MoEs live at aifoss.dev.
The 512GB Mac Studio M5 Ultra still isn’t orderable. Apple’s “late October” window for the one single-box machine that could hold Mistral Large 4 is now days from expiring, the configurator still stops at 256GB, and pricing remains unannounced — with one forum megathread reporting slippage to late January. Muse Spark’s flashiest benchmark (24.99 tok/s tensor-parallel across two of them) stays a benchmark of hardware you cannot buy. We’ll cover orders opening the day it happens.
Qwen3.8-Omni-Flash stayed closed. A month of re-checking since September’s edition: still API-only, still no weights. The pattern holds — treat “Omni” as a hosted product line now.
Step 5 Preview shipped two days late — but shipped. The October 15 date slipped; official FP8 and BF16 checkpoints were confirmed live October 17. The only measured local run remains the 4× DGX Spark cluster at 21–30 tok/s on a ~$18,800 pool — reality-check territory, not a recommendation.
What to actually buy
Prices as of October 2026, all verified in the linked guides:
| Your situation | The machine | Price | Where |
|---|---|---|---|
| Want the October 24GB stack (Qwen3.8-27B + Bolt) tonight | Used RTX 3090 24GB | $1,386–$1,450 | Check price |
| Starting out — want Mellum 2.1 / MiMo 9B for coding | Used RTX 3060 12GB | $260–$296 | Check price |
| Want the 93–97GB MoE class (GLM-5.3-Flash, DS4, Super VL) at home | GMKtec EVO-X2 128GB | $3,499–$3,649 | Check price |
| Eyeing Bolt’s DFlash speedup | RTX 5090 32GB | from $4,299 — Micro Center in-store only; online third parties $6,500+ | Call ahead, don’t pay scalpers |
| Undecided — test any of this before buying | Rented 3090 | from $0.07/hr | Vast.ai |
The used 3090 row is $43–$107 higher than it was in September’s edition — the card has climbed every month this year, and October’s releases keep adding reasons to want 24GB, not fewer. The 3060 row is the one genuinely cheap ticket this month’s news created. The 5090 row is there for completeness: first-party online stock collapsed in September and hasn’t returned.
FAQ
Should I replace Qwen3.8-27B with Nemotron Bolt on my 24GB card? No — add, don’t replace. Bolt’s own vendor card scores it marginally below Qwen3.6-35B-A3B on general benchmarks; its win is long context (262K tested, Mamba-cheap KV). Keep the 27B for quality, pull Bolt when a task genuinely needs 128K+.
Is an official 2-bit GGUF actually usable, or is this marketing? Bolt is the test case: NVIDIA published its benchmark table at IQ2_XXS, so the scores you compare are the 2-bit scores — no silent quality cliff between the marketing and your download. That’s the real shift; whether other vendors benchmark their 2-bit artifacts as honestly is worth checking per release.
Mellum 2.1 or the MiMo 9B distill for a 12GB card? Try MiMo’s distill first: it’s smaller (5.84GB), MIT, and loads in tooling you already have. Mellum’s SWA layout handles very long context more gracefully. Both benchmark sheets are vendor-only right now — trust your own repo over either number.
Does GLM-5.3-Flash work in Ollama now that llama.cpp merged support?
Not officially yet. Mainline llama.cpp loads current GGUFs, but Ollama and LM Studio need to ship builds that include the arch, and old community GGUFs converted under the early arch name may still fail. Direct llama-server use works today.
Whatever happened to the Ollama qwen3.8 MTP slowdown?
Closed, not reproduced — a collaborator measured 107–115 tok/s with 0.94 draft acceptance across every version from 0.33.2 to 0.34.2. If you set draft_num_predict 0 as a precaution, remove it.
Sources
- llama.cpp b11709 release notes — Kolibri-1 support, PR #30673 — GitHub
- llama.cpp releases index (Oct 17 builds: MMVQ low-bit kernels, HF model browser) — GitHub
- GLM-5.3-Flash support PR #27754, closed Oct 1 “support is already in” — ggml-org/llama.cpp
- GLM-5.3-Flash llama.cpp guide and UD-IQ3_XXS sizing — Unsloth docs
- Ollama issue #18541: qwen3.8:27b MTP regression, closed with cross-version bench — GitHub
- MiMo-V2.6-Pro model page and Intelligence Index placement — Vals AI
- Xiaomi MiMo-V2.6 release overview, family specs, MIT license — eesel
- MiMo-V2.6 9B distill: 5.84GB Q4_K_M, SWE-bench deltas — AlphaSignal
- MiMo-V2.6-Distill-Qwen-9B benchmark table vs Qwen3.5-9B base — Featherless
- Nemotron 3.5 Bolt llama.cpp integration PR #30520 — ggml-org/llama.cpp
- Qwen3.8-27B MTP speculative decoding bench, RTX 3090 — HackMD/thc1006
- Muse Spark Open on 2× M5 Ultra via MLX TB5 RDMA — MacRumors
- Step 5 Preview measured 4× DGX Spark run — local-ai-registry PR #172
- RTX 3090 price, October 2026 — GPU Poet
- RTX 5090 EU street prices exceed €5,000 — Guru3D
Last updated October 18, 2026. Prices and specs change; verify current rates before purchasing.
Recommended Gear
- RTX 3090 24GB (used) — the card the October 24GB stack is built on
- RTX 3060 12GB (used) — the cheap ticket into the new 12GB coding tier
- GMKtec EVO-X2 128GB — the door into the 93–97GB MoE class
Was this article helpful?
Thanks for the feedback — it helps improve future articles.
Need hands-on help?
I offer 1-on-1 technical consulting for local AI setup, GPU selection, and AI coding tool configuration — same topics covered on this site.
Book a session — $49 / hour →Get the numbers before you buy
New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.