Aug 17, 2026

Nemotron 3.5 Lightning on Consumer GPUs in 2026: The 25GB Q4 Catch, and Real Speeds From RTX 5090 to M4 Max

By RunAIHome Team · 13 min read

nemotronnvidialocal-llmmoertx-5090rtx-3090ollamaagentic-aimamba

TL;DR: NVIDIA’s Nemotron 3.5 Lightning (released August 11, 2026, permissive OpenMDW-1.1 license) is a 30B MoE with 3B active parameters, a hybrid Mamba-2 design, and a 1M-token context — built to be the fast “grunt work” layer of an always-on agent, not the smartest model you own. The catch for this site’s readers: the standard Q4_K_M GGUF is 25GB, which skips right past every 24GB card. It’s measurably the throughput king of the 30B class, and measurably not the accuracy king.

	Nemotron 3.5 Lightning	Nemotron-Cascade 2	Muse Glimmer 30B
Best for	Always-on agents, long contexts	Interactive coding on 24GB	Tool-calling, multimodal agents
Params	30B MoE, 3B active	30B MoE, 3B active	30B dense
4-bit footprint	25GB Q4_K_M (32GB card)	18.2GB IQ4_XS (fits 24GB)	17.3GB Q4_K_M (fits 24GB)
Measured speed	~123 tok/s (RTX 5090, Ollama)	187 tok/s (RTX 3090, IQ4_XS)	75 tok/s (RTX 4090)
The catch	Q4 doesn’t fit 24GB cards	24GB VRAM floor is firm	DFlash drafter needs +5.11GB

Honest take: on a 32GB RTX 5090 or a big unified-memory machine, Lightning is the best long-context agent workhorse you can run right now — its speed doesn’t sag as the context fills, and nothing else in the class does that. On a 24GB card, drop to the 19.8GB Q3_K_M or just keep running Nemotron-Cascade 2, which is faster there anyway.

NVIDIA shipped Nemotron 3.5 Lightning on August 11, 2026 as the opening act of its “Month of Local AI” push, and unlike most NVIDIA model launches, this one is aimed at hardware you might actually own. The pitch, per NVIDIA’s technical blog, is up to 4× the token generation speed of similar-sized open models and 30% faster end-to-end task completion — for the repetitive layer of agent work: tool calls, retrieval, validation, formatting, summarization. Before you pull anything, check your card against the sizes below or in the VRAM calculator, because the quantization lineup has a genuine surprise in it.

What Lightning actually is

Three architectural facts explain everything else in this article, all from NVIDIA’s model card and launch blog:

30B total, 3B active — but hybrid. Like Nemotron-Cascade 2 before it, Lightning is a sparse MoE that reads only ~3B parameters per token, so decode speed looks like a 3B model while capability looks closer to a 30B. New this generation: the MoE layers are interleaved with Mamba-2 layers and only a few attention layers. Mamba-2 is a state-space design whose per-token cost doesn’t grow with context length — which is why this model’s generation speed stays flat when a transformer’s would collapse.
1M-token context. The Ollama build ships with a 1M context window. That’s not a typo, and it’s the always-on agent angle: an agent that runs for hours accumulates enormous context, and Lightning is built to keep working through it. Practically, Ollama’s launch post suggests planning for 64K–128K on a 32GB card — KV cache and Mamba state still cost VRAM even when the architecture is frugal with compute.
MTP plus two draft models. Lightning ships with multi-token prediction and dedicated DSpark and DFlash drafters for speculative decoding, per the-decoder’s analysis. Same technique family as the llama.cpp speculative decoding setups we covered in July — the output is identical, it just arrives faster.

The license is OpenMDW-1.1, a permissive open license — no user caps, no regional exclusions like the ones that gutted MiniMax H3’s local story. And to answer the naming confusion up front: this is a new model, not a rebadge. It’s the successor to Nemotron 3 Nano 30B in NVIDIA’s lineup, sitting below Nemotron 3 Ultra and beside the coding-tuned Nemotron-Cascade 2.

The 4× claim, unpacked

NVIDIA’s headline is “up to 4× the output speed of similar-sized models,” and the fine print matters. The comparison set is similar-sized open models — the Qwen family is the named baseline — and the speedup comes from three stacked techniques: 3B-active MoE, NVFP4 quantization, and MTP speculative decoding with the bundled drafters. Independent testing broadly backs the throughput story:

Artificial Analysis measured nearly 670 tokens/second on a pre-release DeepInfra endpoint serving the final NVFP4 weights — the fastest model in its class in their testing. The median across API providers now sits at ~297 tok/s, with DeepInfra at 593.6 and Fireworks (BF16) at 497.5.
On PinchBench, NVIDIA’s 10,000-task agent benchmark, Lightning reaches ~86% accuracy while finishing the run 30% faster than Qwen3.6 35B-A3B at similar accuracy.

Now the honest part. On NVIDIA’s own benchmark table, Lightning trails Qwen3.6 35B-A3B on 13 of 14 accuracy rows. Artificial Analysis scores it 24 on their Intelligence Index — roughly on par with GPT-OSS-120B at about a quarter of the total parameters, which is genuinely impressive efficiency, but nobody’s frontier model. The published BF16 scores: 51.56 on SWE-bench Verified, 75.44 on GPQA Diamond, 81.94 on MMLU Pro. Compare that SWE-bench number to Qwen3.8-27B’s 77.2%-class coding results and the division of labor is obvious: Lightning is not the model you hand a hard bug. It’s the model that executes ten thousand small steps without making you wait.

The VRAM map — and the 25GB problem

Here’s where the queue-jumping happens. Every verified size, from Unsloth’s GGUF repo, bartowski’s repo, and the Ollama library tags:

Build	Size	Fully fits on	Notes
Q3_K_M (bartowski)	19.8GB	24GB cards	The realistic 3090/4090 option
NVFP4 (official)	~20GB min	RTX 50-series, 24GB+	NVIDIA’s stated 4-bit minimum; Blackwell-optimized
MXFP4_MOE (Unsloth)	23.2GB	32GB cards	Too tight on 24GB once KV cache loads
Q4_K_M / Ollama default	25GB	32GB cards	What `ollama run` pulls
MLX 4-bit (Ollama)	23GB	32GB+ unified Macs	256K context tag
Q8_0 (Unsloth)	35GB	Nothing consumer	Multi-GPU or unified memory
MXFP8 (Ollama MLX)	34GB	64GB Macs	—
BF16	~60GB	Nothing consumer	Fine-tuning source

Read that middle row again. Nemotron-Cascade 2’s IQ4_XS came in at 18.2GB and left a used RTX 3090 with 5–6GB of cache headroom. Lightning’s standard 4-bit GGUF is 25GB before a single token of context — the Mamba-2 layers, MTP head, and vocabulary all add weight that the “30B” label doesn’t advertise. A 24GB card cannot hold it.

The problem shows up exactly the way it did with Muse Glimmer’s drafter: you run ollama run nemotron-3.5-lightning on a 3090, the pull succeeds, generation starts — and it’s inexplicably slow, because Ollama silently split the overflow to CPU. Check it yourself:

$ ollama ps
NAME                             SIZE     PROCESSOR          UNTIL
nemotron-3.5-lightning:latest    27 GB    12%/88% CPU/GPU    4 minutes from now

Anything other than 100% GPU on that PROCESSOR column means you’re spilling. Two real fixes on a 24GB card:

Drop to Q3_K_M (19.8GB). Pull bartowski’s quant into llama.cpp or import it as an Ollama modelfile. You lose some quality versus Q4 — on a model already positioned below Qwen on accuracy, that’s a real cost, but it keeps the whole model on-die with ~4GB for context.
Offload the experts, keep the attention. Because only 3B parameters are active per token, llama.cpp’s MoE offload (--n-cpu-moe, or -ot ".ffn_.*_exps.=CPU") parks cold experts in system RAM at a much smaller speed penalty than dense-model spill. You’ll need the RAM — budget 32GB system memory — and it still won’t match a card that fits the whole file.

Or accept the model’s actual target: this is the first consumer-relevant NVIDIA release that treats 32GB as the entry tier — an RTX 5090, a DGX Spark, a Strix Halo box, or a 36GB+ Mac.

Real speeds, real hardware

Independent numbers from the first week, not NVIDIA’s marketing deck:

Hardware	Config	Measured speed	Source
RTX 5090 32GB	Ollama, 25GB GGUF, 64K ctx	~123 tok/s avg on short tasks	KGP Talkie
MacBook M4 Max 64GB	6K-token repo prompt	~70 tok/s, 6s prefill	Tom Greenwald’s test
DGX Spark	single-stream	~71 tok/s	Kubesimplify
DeepInfra H100s (NVFP4)	API, batched	593.6 tok/s	Artificial Analysis

The M4 Max comparison is the one that sells the architecture. Same machine, same repo, same 6,000-token prompt: dense Qwen 3.8 27B took 32 seconds of prefill and generated at ~15 tok/s; Lightning took 6 seconds and generated at ~70 tok/s. That’s the MoE-plus-Mamba dividend in one data point — and it compounds, because in KGP Talkie’s RTX 5090 testing, Lightning was the only model in the 30B group whose speed did not degrade as the 64K context filled. Dense transformers slow down as the window loads up; the Mamba-2 hybrid holds flat. For a chat session that never exceeds 8K, that’s irrelevant. For an agent grinding through hour twelve of a monitoring loop, it’s the whole product.

One caveat worth stating plainly: the eye-catching 670 tok/s figure is datacenter NVFP4 serving with speculative decoding fully engaged, not what your card will do. The llama.cpp/GGUF path doesn’t yet use the DSpark/DFlash drafters the way NVIDIA’s own stack does, so local speeds are “fast 3B-active MoE” fast — think 2–4× a comparable dense 27B on the same silicon, consistent with what we found when we broke down how MoE and speculative decoding actually stack.

Running it

Ollama is the shortest path (v0.32+; the library tag landed at launch):

$ ollama run nemotron-3.5-lightning

That pulls the 25GB default. The tag list as of August 17, 2026: latest/30b/30b-a3b (25GB, 1M context), 30b-a3b-mlx (23GB, 256K, Apple Silicon), 30b-a3b-mlx-bf16 (66GB), 30b-a3b-mxfp8 (34GB). Ollama’s new agent launcher also wires it straight into coding agents:

$ ollama launch claude --model nemotron-3.5-lightning

— which starts Claude Code against the local model as backend. The same launcher supports opencode, hermes, and openclaw. If you’re choosing an agent front-end for it, aicoderscope.com covers that side of the stack.

For llama.cpp, use Unsloth’s recommended sampling: --temp 0.6 --top-p 0.95 --min-p 0.01. On vLLM, note that day-0 support initially pointed at a nightly image, and NVIDIA’s own NIM quick-start pairs the nemotron_v3 reasoning parser with the qwen3_coder tool parser — if your tool calls come back malformed, mismatched parser flags are the first thing to check.

No 32GB GPU and no urge to buy one? A cloud pod is the cheap way to find out whether Lightning fits your agent stack before committing hardware money — RunPod rents 32GB+ cards by the hour.

Which hardware makes sense for it

Own a 24GB card already? Run Q3_K_M if you specifically want Lightning’s long-context behavior; otherwise Nemotron-Cascade 2 at IQ4_XS remains the better 24GB citizen — 18.2GB, verified 187 tok/s on a used RTX 3090, and stronger coding scores. A used 3090 runs $1,050–$1,500 on eBay in August 2026 (fair-price trackers put the typical ask at $1,201–$1,299), and nothing about this launch changes its value-king status at that tier.

Buying for this model? The RTX 5090’s 32GB is the first consumer card where the full Q4_K_M loads with context headroom — this launch is quietly the best argument yet for the 32GB tier we mapped in the 32GB VRAM guide. At $3,000+ it’s a lot of money to run a 3B-active model, but it’s also the only card that runs everything in this article’s tables. An RTX 4090 buys you nothing over a 3090 here; both are 24GB and both face the same Q3-or-offload choice.

Unified memory is the sleeper pick. A 64GB M4 Max ran it at ~70 tok/s with 6-second prefill on a real repo prompt, and the MLX tags are first-party. For an always-on agent box that idles at laptop wattage instead of a 400W tower, that math is genuinely competitive — the same capacity-over-bandwidth trade we saw with Strix Halo machines, except this model’s 3B active parameters are exactly the workload unified memory handles well.

FAQ

Is Nemotron 3.5 Lightning the same as Nemotron-Cascade 2? No. Both are 30B-A3B MoEs, but Lightning is a new hybrid Mamba-2 architecture with a 1M context and MTP drafters, positioned as an agent execution model. Cascade 2 is the coding-tuned pure-transformer MoE from June. They also quantize very differently: Cascade 2’s 4-bit fits a 24GB card, Lightning’s doesn’t.

Can I run it on a 16GB card? Not usefully. The smallest sane quant is nearly 20GB; a 16GB card would spill half the model to CPU. On 16GB, Gemma 4 26B-A4B QAT and GPT-OSS 20B remain the picks.

Is it better than Qwen3.6 35B-A3B? Faster, yes — same accuracy, no. NVIDIA’s own table shows Qwen ahead on 13 of 14 rows. Lightning wins when tokens-per-second and long-context stamina matter more than the last few benchmark points: agent loops, batch processing, summarization pipelines.

Does the 1M context actually work locally? The weights support it, but VRAM doesn’t come free — Ollama’s guidance is 64K–128K planned context on a 32GB card. The differentiating feature isn’t the ceiling, it’s that generation speed stays flat as the window fills.

What does “OpenMDW-1.1” let me do? It’s a permissive open license — commercial use, fine-tuning, and redistribution are allowed, with none of the regional exclusions or revenue caps that hedge other 2026 releases.

Sources

Last updated August 17, 2026. Prices and specs change; verify current rates before purchasing.

Recommended Gear

Products linked in this article:

RTX 5090 — the 32GB card that fits Lightning’s full Q4_K_M with context headroom
RTX 3090 — used 24GB value pick; runs Q3_K_M, or Nemotron-Cascade 2 at full quality
RTX 4090 — same 24GB constraint as the 3090; buy only if you want its bandwidth for other models

Was this article helpful?