Ollama v0.40.6 Finally Reads Q2_0 GGUFs: A 27B Model on a $260 12GB Card, One Pull, No Fork (2026)

By RunAIHome Team · 14 min read
ollamaquantizationternaryggufvramgpulocal-llm

TL;DR: Ollama v0.40.6 (October 16, 2026) added the Q2_0 tensor type to its GGUF reader, and a day later the ternary Bonsai 2 family quietly appeared on the official Ollama library — so a 27B-class model is now one ollama pull away from any 12GB card, at a measured ~34 tok/s on an RTX 3060. The catch: Q2_0 quality is real but uneven, and agentic coding is where it drops hardest.

bonsai-2:27b (Q2_0)Qwen3.8-27B GSQ-RCO IQ3_SQwen3.8-27B Q4_K_M
Best for27B reasoning on a 12GB card, zero setupNear-base quality on a 16GB cardFull quality, long-context headroom
Weights on disk8.3GB (one pull)11.8GB16.8GB
The catch~75% retention on agentic coding benchmarksManual GGUF import, 16GB minimumNeeds a 24GB card ($1,399+ used)

Honest take: If you own a 12GB card, pull bonsai-2:27b tonight — it’s the first 27B-class model that runs at reading speed on hardware this cheap with stock tooling. But if you’re buying a card for local AI, 2-bit doesn’t change the advice: the used RTX 3090 still buys you every model at full quality, not one model at 98%.

Three weeks ago, running Ternary Bonsai 2 27B meant compiling PrismML’s llama.cpp fork and hoping the GGUF you grabbed matched the kernels you built. As of October 16, it means upgrading Ollama. That’s the whole story — and it’s a bigger deal than the one-line changelog entry (“gguf: add the q2_0 tensor and file types”) suggests, because it moves an entire class of 2-bit models from “enthusiast science project” to “default tooling.”

Here’s what actually changed, what runs on which card, the measured speeds, and where the quality cost actually lands.

What v0.40.6 actually changed (it’s smaller than you’d think)

The fix is two pull requests and one sentence of engine work:

  • llama.cpp gained the Q2_0 quantization type upstream earlier in 2026 — a group-of-64 format that stores 64 weights in 18 bytes, which works out to 2.25 bits per weight. Ollama has been bundling a Q2_0-capable llama.cpp (build b11669) since v0.40.5.
  • But Ollama parses GGUF files with its own Go reader before the engine ever sees them, and that reader didn’t recognize tensor type 42 (GGML_TYPE_Q2_0) or file type 41. Any GGUF containing a Q2_0 tensor failed at ollama create or model load with a tensor-size overflow error — even though the bundled engine could run it. A community PR (#18443) flagged this in September; Ollama founder Jeffrey Morgan’s own #18967 merged the fix on October 15, and v0.40.6 shipped it on October 16.

In the merged PR’s own testing, Bonsai 2 at 2B, 4B, 9B, and 27B all “create and run” — thinking, vision, and tool calls included — where v0.40.5 fails on the identical files. The same fix closed issue #18817, where ISTA-DASLab’s Q2_0 build of the 125B Qwen3.8-Flash-Next MoE hit the identical import wall.

Check what you’re running before anything else:

$ ollama -v
ollama version is 0.40.6

Anything below 0.40.6 and every Q2_0 file in this article fails to load. The upgrade is the standard installer re-run on Windows/macOS or curl -fsSL https://ollama.com/install.sh | sh on Linux.

What Q2_0 is — and the one thing it isn’t

Q2_0 is a container, not a magic wand. The models shipping in it today were made 2-bit-safe by their authors, through one of two routes:

Ternary quantization-aware training. PrismML’s Bonsai 2 line stores every weight as -1, 0, or +1 — a property trained in with QAT and distillation from the Qwen3.8 base models, not bolted on afterward. Our ternary deep dive covers the mechanism; the short version is that ternary survives 2 bits because the model never had more than three weight states to lose.

Calibrated post-training quantization. ISTA-DASLab — the lab behind GPTQ — ships GSQ-RCO quants of the actual Qwen3.8 models: learned quantization grids per tensor, plus an optimizer that assigns each tensor a type under a size budget. Their Qwen3.8-Flash-Next Q2_0 (2.40 bits/weight, 66.4GB) is built for speed — the card reports 3.4× the prompt throughput of IQ2_XS at near-identical quality (89.07 vs 89.16 task average).

What Q2_0 is not: a flag you point at your favorite model. Running llama-quantize to a 2-bit type on ordinary FP16 weights completes without errors and produces garbage — llama.cpp #15193 documents it, and nothing in v0.40.6 changes it. You run the 2-bit files labs publish; you keep quantizing your own models to Q4_K_M like before.

Bonsai 2 is now in the official Ollama library — quietly

Around October 17, PrismML’s models appeared at ollama.com/library/bonsai-2 with no announcement — OrcaRouter spotted the listing before the vendor said a word. Four sizes, each with a default tag and a -q2_0 alias pointing at the same digest (the Q2_0 packing is the default), 256K context, vision and tools enabled:

TagDownloadFits comfortably on
bonsai-2:2b911MBAnything, including phones-adjacent hardware
bonsai-2:4b1.6GB4GB+ VRAM
bonsai-2:9b3.2GB6-8GB VRAM — this is the 8GB-card pick
bonsai-2:27b8.3GB12GB VRAM

The download sizes run larger than PrismML’s quoted model-only numbers because the Ollama blobs bundle the vision projector and metadata. Plan around the numbers above.

One pull, no fork, no CMake:

$ ollama run bonsai-2:27b "Explain PCIe lane bifurcation in two sentences."

First run pulls 8.3GB; after that it loads like any other model. Note what this does not cover: the PrismML fork’s PQ2_0 (7.21GB) and PTQ1_0 (5.95GB) packings are different, denser layouts that still require the fork — community Ollama mirrors of those files still fail or emit gibberish on stock builds. If you grabbed a third-party pq2_0 tag and got garbage, that’s the wrong packing, not a broken model.

Measured speed: a $260 card at reading speed on a 27B

The numbers below are fork-era community measurements of the same ternary weights — decode speed is set by how many bytes cross the memory bus per token, so the packing’s runtime matters less than the format’s size. Sanity-checked against each card’s bandwidth ceiling (bandwidth ÷ ~7.4GB of weights read per token for a dense 27B at 2.25 bits/weight), every row lands in the plausible 50-75% band:

GPUBonsai 2 27B decodeCeiling check
RTX 3060 12GB (360 GB/s)~33-34 tok/s measured (HF discussion, video test)70% of 48.6
RTX 5060 Ti 16GB (448 GB/s)37 tok/s measured61% of 60.5
RTX 4070 12GB (504 GB/s)37.2 tok/s measured55% of 68
RTX 4090 24GB (1,008 GB/s)91 tok/s measured67% of 136
RTX 5090 32GB (1,792 GB/s)129.9 tok/s measured54% of 242

Two things worth noticing. First, the absolute story: a used RTX 3060 12GB — $260-$296 on the used market as of late September 2026 — decodes a 27B-class model at 3-4× reading speed. Before Q2_0, that card’s ceiling was 12-14B models at Q4.

Second, the shape of the table: the RTX 5060 Ti has 24% more bandwidth than the 3060 but only decodes ~9% faster, and the faster cards sit progressively further under their ceilings. That HF discussion thread asking “why so little difference?” has the honest answer: 2-bit unpacking costs compute per byte, so low-bit formats lean less purely on bandwidth than Q4 does — the same physics that makes decode bandwidth-bound at 4 bits starts sharing the bill with the shader cores at 2.25. Cheap cards close the gap; expensive cards waste headroom.

The VRAM fine print: the KV cache didn’t get the memo

Ternary compresses weights, not context. Measured totals for the fork’s 5.95GB PTQ1_0 packing run about 6.8 GiB at 4K context, 8.6 GiB at 32K, and 22.9 GiB at the full 262,144-token window with an f16 cache. The Ollama Q2_0 build’s weights are roughly 2.3GB heavier, so figure — this is arithmetic on those measurements, not a new measurement — roughly 9 GiB at 4K and 11 GiB at 32K.

On a 12GB card, that means the 256K context on the model card is theoretical: you’re realistically running 8K-32K, which covers chat and single-file coding and does not cover whole-repo dumps. If Ollama starts evicting the model mid-session, that’s VRAM pressure from context, and the keep-alive and loading behavior is tunable. Check your exact model + context combination in our VRAM calculator before blaming the card.

On an 8GB card, the 8.3GB 27B doesn’t fit, full stop. Your stock-Ollama play is bonsai-2:9b; your fork play is the 5.95GB PTQ1_0, which loads with almost nothing left for context. The 9B is the sane answer.

Quality: the part the YouTube titles skip

The vendor numbers are good and the fine print matters. PrismML’s whitepaper scores Bonsai 2 at 84.78 across a 14-benchmark suite against 86.32 for the full-precision Qwen3.8-27B — the “98.2% retention” claim checks out on average. But the same whitepaper shows retention falling to roughly three quarters of the base model’s score on Terminal-Bench 2.1 and SWE-bench Verified. The capability 2-bit gives up first is exactly the multi-step agentic coding that’s been the year’s main event.

Independent signals agree. Aggregator BenchLM places Ternary Bonsai 2 27B at 50.8/100 overall (#113 of 230 tracked models), with τ²-bench at 80.2% but Terminal-Bench 2.1 at 52.8%. And one developer ran four 2-bit 27B-class builds through 20 multi-step bug fixes on a single RTX 3060: Bonsai 2 27B solved 13/20, ISTA’s GSQ-RCO 2-bit of the real Qwen3.8-27B solved 16/20, and two other community 2-bit conversions managed 10/20 and 7/20. One tester, one benchmark — but the spread is the lesson. “2-bit” is not one quality tier; the conversion method matters more than the bit count.

That GSQ-RCO result deserves a pointed note for 12-16GB owners. ISTA-DASLab’s Qwen3.8-27B-GSQ-RCO-GGUF ships IQ2_XS at 8.4GB (2.50 bits/weight, scoring above the BF16 baseline on zero-shot suites per the model card) up to the recommended IQ3_S at 11.8GB, which matches the base model exactly on AIME25 and LiveCodeBench v6. Those are standard GGUF types Ollama could import even before v0.40.6 — on a 16GB card, IQ3_S is arguably the better 27B than the ternary, and on a 12GB card the IQ2_XS is the direct Bonsai competitor the bug-fix test says to prefer for coding. The 2-bit conversation Ollama’s update started is bigger than one model family; October’s open-model roundup shows vendors converging on official 2-bit artifacts across the board.

For a feel of where the floor is: one hands-on reviewer watched Bonsai 2’s reasoning trace correctly locate a planted bug in a multi-service Docker app, then talk itself out of the fix and edit an unrelated file instead. 27B-class reasoning at 2.25 bits is real; it is not the same model as the 16.8GB Q4_K_M, and anyone telling you otherwise is reading the average, not the tails.

If something breaks: the three failure modes

“Tensor size overflow” / unsupported-tensor error on pull or create. You’re on Ollama ≤0.40.5 reading a file with Q2_0 tensors. This also bit parts of the Nemotron Bolt 74B GGUFs. Upgrade to 0.40.6+, confirm with ollama -v, retry. LM Studio’s pinned GGUF library still can’t read the type as of this writing — Ollama or llama.cpp are your runtimes.

Gibberish output from a community Bonsai tag. You pulled a PQ2_0/PTQ1_0 fork packing through a stock runtime. The official bonsai-2 library tags are the stock-compatible Q2_0; the denser packings still need PrismML’s fork build.

Model loads, then evictions or OOM mid-conversation. Context, not weights — the KV cache at long contexts outgrows what the 2-bit weights saved. Drop num_ctx to 8K-16K on a 12GB card.

What to actually buy

Prices as verified against our tracked street-price index (RTX 3060: September 27; RTX 3090: October 9, 2026):

Your situationThe movePriceWhere
Already own any 12GB cardollama pull bonsai-2:27b, spend nothing$0—
Want the cheapest ticket to 27B-class local AIUsed RTX 3060 12GB$260-$296Check price
Want 27B at full Q4_K_M quality + context headroomUsed RTX 3090 24GB$1,399-$1,450Check price
Not sure 2-bit quality fits your workloadRent a 3090 for an evening and testfrom $0.07/hrVast.ai

The honest framing of that first row: Q2_0’s real achievement is making the hardware you already own better. The used RTX 3090 remains the buy-once answer — it runs the full-quality 27B at 41 tok/s, the ternary build faster than any 12GB card, and everything else in the GPU buying guide without waiting for a lab to bless your model with a 2-bit conversion. If your local stack feeds a coding assistant, that quality gap at the agentic tail is exactly where you’d feel it — pair whatever you run with a sane local-backend setup and test on your own repos before trusting it. For the broader Ollama picture beyond this one release, our sister site keeps a living Ollama review.

FAQ

Can I quantize my own models to Q2_0 now? No. The type being readable doesn’t make arbitrary weights survive it — naive 2-bit quantization of a normal model produces garbage output. The usable Q2_0 files come from labs that trained ternary weights in (PrismML) or ran calibrated grid-learning pipelines (ISTA-DASLab).

Does this make the 125B Qwen3.8-Flash-Next a 12GB model? No — its Q2_0 build is 66.4GB and wants 64GB+ of system RAM with CPU/SSD offload, the territory our Strata engine guide covers. v0.40.6 removed Ollama’s import error for those files; it didn’t shrink them.

Is bonsai-2:27b better than a 9B at Q4? For reasoning breadth, yes — 27B-class knowledge survives the bit cut better than you’d guess, and 13/20 on a multi-step coding bench beats what 9B dense models typically manage. For tasks where its specific weaknesses bite (agentic multi-step coding, long-context work past 32K on a 12GB card), a dense 9-14B at Q4 can be the more predictable tool.

Why is my speed lower than the table? The measurements above are Linux/CUDA decode speeds at moderate context. Long prompts shift time into prefill, Windows typically lands a few percent lower, and a PCIe-starved slot or partial CPU offload craters everything — ollama ps should show 100% GPU.

Sources

Last updated October 19, 2026. Prices and specs change; verify current rates before purchasing.

Products linked in this article:

Was this article helpful?

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.