NVIDIA Nemotron 3.5 Bolt 74B for Local AI: The First 74B Agent Model Shipped to Fit a 24GB Card (2026)

By RunAIHome Team · 14 min read
nemotronnvidialocal-llmmoertx-3090rtx-5090ollamallama-cppagentic-aiquantization

TL;DR: NVIDIA released Nemotron 3.5 Bolt 74B-A9B on October 15, 2026 — a 74B-parameter MoE shipped natively as a 22.02GB 2-bit GGUF that fits a 24GB card, under the permissive OpenMDW-1.1 license. It’s the largest model you can run on a used RTX 3090 without CPU offload. The catch: its benchmark score lands a hair below Qwen3.6-35B-A3B, which already fit your card.

Nemotron 3.5 Bolt 74B-A9BQwen3.6-35B-A3BNemotron 3.5 Lightning
Best forLong-context agents on 24GBAll-round quality on 24GBLong-context agents on 32GB+
Params74B MoE, 9B active35B MoE, 3B active30B MoE, 3B active
File that fits 24GB22.02GB native IQ2_XXS21GB Q4_K_MNone (Q4 is 25GB)
PinchBench (agent tasks)87.82 (at 2-bit)88.07 (at BF16)85.37 (at BF16)
The catchSpec-decode drafter needs 32GBSlower per token (more active GB)Skips 24GB cards entirely

Honest take: if you own a 24GB card, pull Bolt this weekend for long-context agent work — its 256K window and Mamba-heavy design do things Qwen can’t. But don’t buy a new GPU for it: at everyday quality it’s a tie with the Qwen3.6-35B-A3B you can already run, and NVIDIA’s “#1 open model in its class” line survives only until you read the second row of its own benchmark table.

NVIDIA shipped two open-weight models on October 15: Nemotron 3.5 Super VL (120B-A12B, multimodal, needs ~70GB+ of memory) and the one this article is about — Nemotron 3.5 Bolt 74B-A9B, which NVIDIA packaged specifically to fit in 24GB (NVIDIA announcement, Wccftech coverage). A third model, the 4GB Bolt E4B for 8GB cards and laptops, follows on October 23.

The interesting part isn’t the parameter count — it’s the packaging. Every previous “can I run a 70B-class model on my 24GB card” story on this site ended the same way: no, not really, 2–5 tok/s with CPU offload. Bolt is the first time a vendor has shipped the 2-bit quantization as the product, quantization-aware, with the benchmark scores measured at 2-bit. Whether that changes what you should buy — mostly it doesn’t, and the one case where it does is below — takes some unpacking.

What NVIDIA actually shipped

Four facts define the model, all from the official Hugging Face repo and the llama.cpp support PR:

  • 74B total, 9B active, hybrid. Like Nemotron 3.5 Lightning before it, Bolt interleaves Mamba-2 layers with a small number of attention layers and sparse MoE blocks (the conversion code calls the architecture nemotron_h_moe). Only ~9B of the 74B parameters are read per token, so decode speed behaves like a 9B model while capability is supposed to behave like something much larger.
  • Native 2-bit, not an afterthought. The official GGUF stores the MoE expert weights as IQ2_XXS tensors and weighs 22.02GB. NVIDIA calls it the “24GB packed model” and publishes its benchmark scores for this file, not for a BF16 original you can’t run. A BF16 checkpoint exists for fine-tuning, but the 2-bit build is the product.
  • 256K context, Mamba economics. The llama.cpp conversion work tests the model at 262,144-token context, and because most layers are Mamba-2 (whose state doesn’t grow with context), the full 256K window costs far less memory than a transformer’s would. Third-party memory modeling puts the whole thing — weights plus full 262K cache — at ~23.3GB (VRAM Calculator breakdown).
  • OpenMDW-1.1 license. Same permissive license as Lightning: commercial use, fine-tuning, and redistribution are allowed. No user caps or regional exclusions.

Support landed fast: llama.cpp PR #30520 (“Add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support”) was merged by ggerganov on October 15, the same day as the release, targeting the v0.7.0 release. Ollama, LM Studio, Unsloth, and vLLM are all on NVIDIA’s supported list — with one LM Studio caveat covered below.

The 24GB fit, in actual numbers

A 22.02GB file on a 24GB card leaves roughly 2GB for everything else, so the fit is real but snug. Here’s how the memory budget breaks down, per the VRAM Calculator analysis (their estimates, built on the published architecture — not NVIDIA’s numbers):

ContextTotal VRAMFits 24GB?
8K~21.1GBYes, with desktop headroom
262K (full), FP16 KV~23.3GBOnly headless — 97% of the card
262K (full), Q8_0 KV~22.6GBYes, barely, headless

Three practical consequences:

  1. On Linux with a desktop environment or on Windows, don’t start at full context. Your display compositor already holds VRAM. Start at 8K–32K and raise it until you hit the wall — check your exact card-and-context combination in our VRAM calculator.
  2. Quantize the KV cache for the attention layers (-ctk q8_0 -ctv q8_0 in llama.cpp) if you want six-figure contexts on a 24GB card. Because only the few attention layers carry a KV cache — the Mamba layers keep a fixed-size state — this costs much less quality than it does on a pure transformer.
  3. 16GB cards are out. There is no smaller official quant, and 2-bit is already the floor — you can’t Q4 your way down from here. If you’re on 16GB, the model NVIDIA wants you to wait for is the Bolt E4B on October 23.

Community re-quants appeared within a day — AtomicChat published a 23.0GB tightened variant and a 26.7GB expanded one with KL-divergence measurements against the original — but for a 24GB card the official 22.02GB file is the sane default.

How fast it should be — and the one number we actually have

NVIDIA published no tokens-per-second figures for Bolt, which launch coverage was right to flag. So let’s do what we always do: compute the physical ceiling, then calibrate against measured models.

Decode speed is memory-bandwidth-bound. Per token, Bolt reads its ~9B active parameters out of a 22.02GB file, which works out to roughly (9/74) × 22.02 ≈ 2.7GB per token. Divide bandwidth by that:

CardBandwidthCeilingOur estimate (~22% of ceiling)
RTX 3090 24GB936 GB/s~347 tok/s~70–80 tok/s
RTX 4090 24GB1,008 GB/s~373 tok/s~75–85 tok/s
RTX 5090 32GB1,792 GB/s~664 tok/s~135–150 tok/s

The 22% calibration comes from the best-measured MoE on this site: Qwen3.6-35B-A3B runs 107 tok/s on an RTX 3090 reading ~1.9GB/token, which is 21.7% of that card’s ceiling. These are estimates, clearly labeled as such, until community benchmarks land — the model is two days old. The one community number so far supports the math: a GGUF maintainer reports ~150 tok/s on an RTX 5090 without speculative decoding, squarely inside our estimated range, though the context length isn’t stated.

Then there are the bundled speculative-decoding drafters, and this is where the 24GB story gets its first asterisk. The llama.cpp integration work measured two drafters for Bolt:

  • LiLiCorr drafter: 83.6% acceptance rate, lifting measured throughput from 128.7 to 224.2 tok/s in the integration harness (hardware unstated — treat the ratio, ~1.74×, as the portable fact, not the absolute numbers).
  • DFlash drafter: better acceptance (91.7%) — but the measured peak was 26.4GiB on a 32GB card. On a 24GB card the DFlash drafter does not fit at any context.

So the headline speculative gains — the same technique family as the llama.cpp speculative decoding setups we covered in July — fully materialize on 32GB cards and unified-memory machines. A 24GB card runs the model; a 32GB card runs it with the good drafter. That’s the first genuine hardware-tier distinction Bolt creates.

The quality fine print: read NVIDIA’s own table

NVIDIA’s launch line is that Bolt is “the #1 open model in its class on PinchBench,” its 10,000-task agent benchmark. The model card’s own numbers, as picked up by third-party comparisons and launch analyses:

  • Bolt 74B-A9B (IQ2, the file you’d run): PinchBench 87.82
  • Qwen3.6-35B-A3B (BF16): PinchBench 88.07

Qwen scores higher. Whatever “in its class” means, the open model most people on a 24GB card actually run beats Bolt on NVIDIA’s own agent benchmark — while Qwen3.6-35B-A3B’s 21GB Q4_K_M fits the same card with more headroom. Artificial Analysis places Bolt at 21 on its Intelligence Index — fourth among open models under 100B parameters. Good, not a new tier.

Where Bolt does look genuinely different is long context. The model card reports an AA-LCR (long-context reasoning) score of 86.43 against Qwen3.6-35B-A3B’s 61.06 — a 25-point gap in the regime where transformer KV caches blow past what a 24GB card can hold and Mamba’s constant-cost state doesn’t. One number from a vendor card, so hold it loosely until independent long-context testing lands, but it’s consistent with the architecture and with what we saw from Lightning: these hybrid models don’t sag as the context fills.

The other open question is what native 2-bit costs. Our canon on quantization is that quality falls off a cliff below Q4 for post-hoc quants. NVIDIA’s counter is that Bolt was trained to be served at 2-bit, and its published scores are measured at 2-bit — the 87.82 already includes the quantization damage, which is exactly how we’d want a vendor to report it. What nobody has yet measured is behavior drift at the edges (code generation, rare languages, very long generations) versus the BF16 checkpoint; early KL-divergence work on the community re-quants is a start.

Running it today, and the three bugs to know about

The shortest path is Ollama pulling straight from the official repo:

$ ollama run hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
pulling manifest
pulling 8f3a2c1d4e5b... 100% ▕████████████████▏  22 GB
verifying sha256 digest
writing manifest
success
>>>

Expect the pull to take a while — it’s a 22GB download — and verify GPU residency afterward with ollama ps (the PROCESSOR column should read 100% GPU; if it says CPU, you’re in the usual Ollama-not-using-GPU territory). For raw llama.cpp, you need a build from October 15 or newer (the merge targeted v0.7.0).

Three real, current issues, all from the integration work and tracker — this is a day-one model and it shows:

  1. OOM at long context on 24GB → quantize the KV cache. If llama-server dies allocating cache above ~128K context on a 24GB card, add -ctk q8_0 -ctv q8_0. That’s the difference between 23.3GB and 22.6GB at the full 262K window — i.e., between crashing and running.
  2. LM Studio can’t read the Q2_0 tensor type yet. LM Studio’s pinned GGUF library predates the new 2-bit type used in parts of these files, per the integration notes. Until LM Studio ships an update, use Ollama or llama.cpp directly.
  3. Mac users: wait. There’s an open Metal segfault for Bolt in ggml_metal_op_mul_mat_id. Third-party MLX conversions surfaced within a day, but the llama.cpp Metal path crashes as of October 16. Give it a week.

What else shipped, briefly

Super VL 120B-A12B is the headline model for NVIDIA — multimodal, 1M context, vision encoder — but not a 24GB story: the smallest build is ~70GB on disk and Unsloth’s run guide uses an 86.9GB Q4 with ~128GB of memory. That’s DGX Spark ($4,699) and 128GB Strix Halo machines territory — a GMKtec EVO-X2 128GB qualifies on capacity, at 256 GB/s of bandwidth, so think ~10 tok/s class, not interactive-fast. Bolt E4B (4GB, October 23) is the one that will matter for 8GB cards; we’ll cover it when it’s real.

What to actually buy

Prices as of October 2026, from this site’s verified price table:

Your situationThe movePriceWhere
Already own a 24GB (or 32GB) cardPull the official GGUF, spend nothing$0—
Want the cheapest card that runs a 74B-class modelUsed RTX 3090 24GB$1,399–$1,450Check price
Want the DFlash drafter + full-context headroomRTX 5090 32GB$3,822–$5,000 (MSRP $1,999)Check price
Want Super VL 120B too, in one quiet boxGMKtec EVO-X2 128GB$3,499–$3,649Check price
Undecided — want to test Bolt on a 3090 firstRented GPU, from $0.07/hrpay per hourVast.ai

The honest ordering: row one covers most readers. Bolt strengthens the case we made in the used RTX 3090 value-king piece — the card keeps gaining capabilities it wasn’t sold with — but it doesn’t yet justify the 5090 premium on its own, because the model you’d run daily on 24GB (Qwen3.6-35B-A3B) still matches or beats it outside long-context work. If agentic long-context is your workload, the 32GB tier now has its first model-shaped reason to exist beyond “bigger quants of the same models.”

For the coding-agent side of this — wiring a local model into Cline or Continue as a BYOK backend — see our sister site’s local coding stack setup, and for the Ollama server side, aifoss.dev’s Ollama review.

FAQ

Does Bolt 74B fit a 16GB card? No. The official file is 22.02GB and it’s already 2-bit — there’s nothing left to shave. On 16GB, run the best models for that tier or wait for Bolt E4B (October 23), which targets ~4GB.

Is this better than Qwen3.6-35B-A3B on a 24GB card? For general and coding use: no — Qwen scores higher on PinchBench (88.07 vs 87.82) and on most public benchmarks, and runs faster per token (1.9GB vs ~2.7GB read per token). For 100K+ context agent sessions: the vendor’s long-context numbers (86.43 vs 61.06 AA-LCR) and the Mamba architecture both favor Bolt, pending independent testing.

Can I run it on a Mac? Not reliably yet via llama.cpp/Ollama — there’s an open Metal crash (issue #30610). Community MLX conversions exist on Hugging Face, but they’re unofficial. Apple-silicon support typically stabilizes within a couple of weeks of a release like this.

Is NVIDIA’s “fits in 24GB alongside other applications” claim true? At short context, yes — ~21GB at 8K leaves room for a desktop. At the full 262K window you’re at 97% of the card headless. “Alongside other applications” and “full context” are mutually exclusive on 24GB.

What’s the catch with native 2-bit? The published scores already include the quantization cost, which is the right way to report it. The unknown is edge-case drift vs BF16 (long generations, code, rare languages) — nobody has measured that independently yet, so treat week-one quality impressions accordingly.

Products linked in this article:

  • Used RTX 3090 24GB — $1,399–$1,450 used (Oct 2026); the cheapest way to run Bolt, and still this site’s VRAM-per-dollar pick
  • RTX 5090 32GB — $3,822–$5,000 street (MSRP $1,999); the tier that unlocks Bolt’s DFlash drafter and full-context headroom
  • GMKtec EVO-X2 128GB — $3,499–$3,649; runs Bolt easily and Super VL 120B slowly

Sources

Last updated October 16, 2026. Prices and specs change; verify current rates before purchasing. Speed figures marked as estimates are bandwidth-ceiling calculations calibrated to measured models, not benchmarks of this model.

Was this article helpful?

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.