JetBrains Mellum 2.1 for Local AI in 2026: 47% SWE-bench From a 12B MoE That Fits a 12GB GPU

By RunAIHome Team · 13 min read
coding-llmjetbrainsmellummoelocal-llmgpu

TL;DR: JetBrains released Mellum 2.1 on October 8, 2026 — a 12B-total / 2.5B-active MoE coding model under Apache 2.0 whose Q4_K_M GGUF is 7.5 GiB, small enough that a used $291 RTX 3060 can run an agentic coding model that self-reports 47% on SWE-bench Verified. The catch: every benchmark number is vendor-reported so far, and there’s an open llama.cpp Metal bug Mac users need to route around.

Mellum 2.1 ThinkingQwen3.6-35B-A3BCodestral 2 22B
Best forAgentic coding on 8–16GB GPUsBest all-round coder if you own 24GBFIM autocomplete, clean license
VRAM + hardware7.5GB Q4_K_M → 12GB card, ~$291 used RTX 306021GB Q4_K_M → 24GB card, ~$1,386+ used RTX 309013.3GB Q4_K_M → 16GB card
The catchVendor-only benchmarks; open Metal NaN bugNeeds a 24GB card that now costs $1,400Dense 22B — slower than both MoEs

Honest take: If you already own a 24GB card, keep running Qwen3.6-35B-A3B — nothing here dethrones it. Mellum 2.1 matters because it’s the first credible agentic coding model for the 12GB tier, where the hardware costs $291 instead of $1,400.

JetBrains spent 2025 shipping Mellum as a 4B autocomplete model that solved about 2% of SWE-bench Verified. Eighteen months later, Mellum 2.1 self-reports 47% — a 23× jump — from a model whose weights quantize down to a 7.5 GiB file. If that number survives independent testing, it changes what the cheapest useful local coding rig looks like. Here’s the hardware math, what you can verify today, and what you can’t.

What Mellum 2.1 actually is

The lineage matters because the name is confusing:

  • Mellum (4B, dense) — JetBrains’ original code-completion model, built for its IDEs, open-sourced April 2025. Autocomplete only; ~2% SWE-bench Verified.
  • Mellum 2 (12B MoE) — released June 2, 2026. Same 12B-total / 2.5B-active architecture as today’s model, in Base, Instruct, and Thinking variants. This is when llama.cpp gained the mellum architecture (PR #23966, merged June 5, 2026).
  • Mellum 2.1 (12B MoE, Thinking) — released October 8, 2026, under Apache 2.0. Architecture unchanged from Mellum 2; JetBrains says the gains come almost entirely from reinforcement-learning post-training. As of mid-October the 2.1 release is the Thinking checkpoint (reasoning trace before the answer); the Instruct and Base variants remain at Mellum 2.

The architecture, from the model config: 12B total parameters, ~2.5B active per token, 64 experts with 8 routed per token, 28 layers, hidden size 2304, grouped-query attention with 32 query heads and 4 KV heads (head dim 128), and a 131,072-token context window. One detail does a lot of quiet work here: three of every four layers use a 1,024-token sliding window, which keeps the KV cache small at long context — more on that below.

VRAM: what each quant needs

Official GGUF sizes from the JetBrains Hugging Face repo (JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF):

QuantFile sizeFits comfortably on
MXFP4_MOE~7.0 GB10–12GB cards
Q4_K_M7.52 GiB (8,071,295,264 bytes)12GB cards
Q6_K~10.9 GB16GB cards
Q8_0~12.9 GB16GB cards (tight), 24GB
BF16~24.3 GB32GB+ or Mac unified memory

JetBrains’ model card lists Q4_K_M at an 88.0% top-token match against the full-precision model and marks it as the recommended quant.

This is a MoE, so remember the rule that trips everyone up: all 64 experts must be resident. The 2.5B active count tells you how fast it decodes, not how much memory it needs. You load the whole 7.5 GiB.

The KV cache math, and why 131K context on 12GB is plausible

With 4 KV heads at head-dim 128 in F16, each layer stores about 2 KB per token. Because 21 of the 28 layers use a 1,024-token sliding window, only the 7 full-attention layers grow with context:

  • 7 full-attention layers × 131,072 tokens × 2 KB ≈ 1.75 GiB
  • 21 sliding-window layers × 1,024 tokens × 2 KB ≈ 0.04 GiB

So Q4_K_M weights (7.52 GiB) plus a full 131K KV cache (~1.8 GiB) lands around 9.3 GiB before compute buffers — tight but plausible on a 12GB card, and comfortable on 16GB. These are our calculations from the published architecture parameters, not measured numbers; they assume llama.cpp’s SWA-aware KV cache (the default in current builds). If you force --swa-full, the cache balloons to ~7 GiB and a 12GB card won’t make it. Start at 8K–16K context and raise it as your VRAM allows — check your exact card and context combination with our VRAM calculator.

How fast it should decode (estimates — no community benchmarks yet)

Nobody has published consumer-GPU tokens/sec for Mellum 2.1 as of October 15, so what follows is arithmetic, clearly labeled. Decode speed is memory-bandwidth-bound: the hard ceiling is bandwidth divided by bytes read per token. At Q4_K_M, the ~2.5B active parameters of the 12B file mean roughly 8.07 GB × (2.5/12) ≈ 1.7 GB read per token. Measured MoE models on this site land well below their ceiling — Qwen3.6-35B-A3B reads ~1.9 GB/token and measures 107 tok/s on an RTX 3090 against a 492 tok/s ceiling, about 22% — so we calibrate estimates at 22–30% of ceiling:

GPUBandwidthCeiling (GB/s ÷ 1.7GB)Estimated real-world
RTX 3060 12GB360 GB/s~212 tok/s~45–65 tok/s
RTX 4060 Ti 16GB288 GB/s~169 tok/s~37–50 tok/s
RTX 5060 Ti 16GB448 GB/s~264 tok/s~55–80 tok/s
RTX 4070 12GB504 GB/s~296 tok/s~65–90 tok/s
RTX 3090 24GB936 GB/s~551 tok/s~115–165 tok/s
RTX 4090 24GB1,008 GB/s~593 tok/s~125–180 tok/s
Mac M4 Pro273 GB/s~161 tok/s~35–50 tok/s

Even the low end of the RTX 3060 estimate is several times faster than reading speed (~7–10 tok/s), and for an agentic model the reasoning trace burns tokens before you see an answer — a Thinking model at 20 tok/s would feel sluggish; at 45+ it’s fine. We’ll replace this table with measured numbers as soon as credible community benchmarks land.

JetBrains’ own throughput claim — roughly 2× the output throughput of Qwen3.5-9B under high load on a single H200 — is a datacenter serving number and tells you nothing about desktop GGUF speed.

The benchmark numbers, and how much to trust them

All vendor-reported, from JetBrains’ October 8 release materials:

  • SWE-bench Verified: 47% — just under the 50% JetBrains measured for Qwen3.5-9B in the same harness.
  • LiveCodeBench v6: 82.0 vs. 75.4 for Qwen3.5-9B — the headline win. Note that Qwen’s own model card reports 65.6 on LiveCodeBench under its own evaluation setup; the two numbers come from different harnesses and shouldn’t be mixed. Compare only within JetBrains’ own table.
  • Qwen3.5-9B still leads on SWE-bench Pro (38.0), AIME, and GPQA Diamond in JetBrains’ own comparison — JetBrains is not claiming across-the-board superiority, which is itself a small credibility signal.

No independent SWE-bench run existed as of October 15. The honest read: a 12B MoE with 2.5B active parameters posting 47% on SWE-bench Verified would have been frontier-model territory in early 2025. It’s plausible — RL post-training on agentic traces is exactly how other 2026 models made the same jump — but treat it as a vendor claim until someone reproduces it. For context on what else you can run, our best local coding LLM guide covers the verified field.

Running it today

llama.cpp has supported the mellum architecture since PR #23966 (merged June 5, 2026). If your build predates that, you’ll get an unknown-architecture error on load — update first. Pull the official GGUF and serve:

# llama.cpp build from June 5 2026 or later
llama-server -hf JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M \
  -c 16384 -ngl 99 --temp 0.6 --top-p 0.95 --top-k 20

Expected output on a 12GB card: the model loads ~7.5 GiB of weights to VRAM, and llama-server reports the sliding-window KV cache allocation. The temperature 0.6 / top-p 0.95 / top-k 20 sampling comes from JetBrains’ published examples.

Ollama can pull the GGUF straight from Hugging Face (ollama run hf.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M); there’s no official Ollama library entry yet, only third-party imports of the older Mellum 2. LM Studio loads the same GGUF through its llama.cpp backend — same minimum-version caveat applies. To wire it into your editor as a local backend, the Continue.dev + Ollama setup on our sister site works unchanged with Mellum as the model.

Known problem: NaN on Apple Silicon (open bug)

Filed October 15, 2026 — llama.cpp issue #30579. On Metal (tested on an M2 Max), the Q8_0, Q6_K, and Q4_K_M quants produce NaN values in perplexity runs. Root cause per the report: activations feeding ffn_moe_down in layer 27 reach ~2.9×10⁵, above the F16 maximum of 65,504, and the Metal matmul kernel stages that input as F16 — so it overflows to inf/NaN. Generated text can look normal because only the final row of logits is used, which makes this nastier than a crash: you may get silently degraded output on specific prompts.

Until the fix lands (the reporter says a fix PR is coming), Mac users have three working configurations:

  • Use the MXFP4_MOE or BF16 quant — those kernels stage as bfloat and are unaffected. MXFP4_MOE at ~7.0 GB is the practical pick.
  • Shrink the micro-batch: -ub 8 avoided the overflow in the report.
  • Run CPU-only (-ngl 0) — correct but slow.

CUDA cards are not implicated in this issue. Separately, if you’re on an AMD Radeon 780M iGPU via Vulkan, an older warmup-hang bug affecting Mellum 2 (issue #24664) was closed in June 2026 — update to a current build and it’s resolved. One more thing to watch: PR #30578 (opened October 15) refines how llama.cpp reads Mellum’s MoE gating parameters from the GGUF, so expect small behavior improvements in builds that include it.

What to actually buy

Prices as of October 2026, all verified this month:

Your situationThe movePriceWhere
Already own a 12GB+ GPU or an Apple Silicon MacRun Mellum 2.1 tonight — $0$0llama-server -hf ... above
Cheapest card that runs an agentic coder wellUsed RTX 3060 12GB~$260–$296Check price
Want Qwen3.6-35B-A3B-class headroom, not just MellumUsed RTX 3090 24GB~$1,386–$1,450Check price
Want to test the model before buying anythingRented RTX 3090, from $0.07/hrpay per hourVast.ai

The interesting row is the second one. Before Mellum 2.1, the honest advice for “I want a local coding agent” was a 24GB card, because the models that could actually do agentic work (Devstral Small 2, Qwen3.6-35B-A3B) needed 17–21GB. If Mellum’s numbers hold up, the entry ticket drops from ~$1,400 to ~$291 — the same kind of tier-collapse that Codestral 2’s 13.3GB Q4 hinted at, but with an agentic model instead of an autocomplete specialist. The used 3090 remains the right call for anyone who wants one card that runs everything in the 12GB-and-up model landscape — see our used RTX 3090 analysis for why it keeps winning on VRAM-per-dollar despite the price creep.

Verdict

Mellum 2.1 is the most interesting small coding model of the fall, and the first time JetBrains’ open-model program has produced something worth building a budget rig around. The Apache 2.0 license is unambiguous, the GGUF sizes are honest, llama.cpp support has been in mainline since June, and the sliding-window architecture makes long-context work on small cards genuinely practical.

What’s missing is independent verification. Every benchmark is vendor-reported, no community tok/s numbers exist yet, and the Metal bug means Mac users should stick to MXFP4_MOE for now. If you have a 12GB card already, pull the Q4_K_M tonight and judge it on your own repos. If you’re buying hardware for it, a used RTX 3060 is a $291 experiment; wait for independent SWE-bench numbers before spending more than that on this model specifically.

FAQ

Does Mellum 2.1 run on an 8GB card? The MXFP4_MOE quant (~7.0 GB) technically loads, but with KV cache and buffers you’d be at 4K-or-less context on 8GB — workable for autocomplete-style use, cramped for agentic sessions. 12GB is the realistic floor.

Is it better than Qwen3.6-35B-A3B? Not if you can run both. Qwen3.6-35B-A3B is a 35B-total MoE that needs a 24GB card; in JetBrains’ own comparison table Mellum trails same-size Qwen3.5-9B on SWE-bench Verified. Mellum’s win is doing agentic coding at all in 7.5 GiB.

Can I use it commercially? Yes — Apache 2.0, no strings. That’s the same license as Codestral 2 post-relicense and more permissive than several Qwen variants.

Why does my Mac output look wrong on some prompts? Likely llama.cpp issue #30579 — F16 overflow in the Metal MoE kernel affecting Q8_0/Q6_K/Q4_K_M. Switch to the MXFP4_MOE or BF16 quant until the fix merges.

Thinking vs Instruct — which do I want? Mellum 2.1 ships as Thinking only (reasoning trace before the answer — better for multi-step agent work, slower to first useful token). For low-latency completion, the older Mellum 2 Instruct is still on Hugging Face.

Sources

Last updated October 15, 2026. Prices and specs change; verify current rates before purchasing. Speed figures in the estimates table are calculated from memory-bandwidth ceilings, not measured — treat them as planning numbers.

Products linked in this article:

Was this article helpful?

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.