NVIDIA Nemotron 3.5 Super VL for Local AI: The 120B Multimodal MoE That Needs 128GB, Not a Blackwell GPU (2026)

By RunAIHome Team · 13 min read
nemotronnvidialocal-llmmoemultimodaldgx-sparkstrix-halomac-studioollamallama-cpp

TL;DR: NVIDIA’s Nemotron 3.5 Super VL (released October 15, 2026) is a 120B-parameter MoE with 12B active that reads text, images, documents, and video — under the permissive OpenMDW-1.1 license. The recommended 4-bit GGUF is 86.9GB, so this is a 128GB-machine model: no consumer graphics card runs it. On a DGX Spark it decodes at a measured 19 tok/s, or ~38 tok/s with NVIDIA’s bundled speculative-decoding draft.

NVIDIA DGX SparkStrix Halo 128GB (EVO-X2 / Framework)Mac Studio M5 Max 128GB
Best forThe measured 38 tok/s recipe, CUDA stackCheapest 128GB that fits the Q4 GGUFFastest bandwidth per dollar at 128GB
Price (Oct 2026)$4,699~$3,449–$3,649~$5,099
What’s verified19 tok/s vanilla / 38 tok/s with draft (measured)Fits 86.9GB Q4; speed estimated ~12–18 tok/sFits 86.9GB Q4; speed estimated ~25–40 tok/s
The catch11-min startup at default settings; one report of shutdowns at 32K+ context256 GB/s bandwidth caps decode speedNo NVFP4 path, no LiLiCorr draft

Honest take: if you already own one of the 128GB unified-memory boxes, Super VL is the most capable vision-language model your machine has ever been able to hold — pull it this week. If you own none of them, buy the cheapest 128GB Strix Halo box only if you need local multimodal agents; for text-only work, Bolt 74B on a used RTX 3090 gets you most of the family’s capability for a third of the money.

NVIDIA shipped three open models on October 15, and this site already covered the small one — Bolt 74B, the 2-bit agent model packaged for 24GB cards. Nemotron 3.5 Super VL 120B-A12B is the flagship of that drop, and it plays a completely different game: it’s multimodal (text, image, document, and video understanding), it carries a 1M-token context window, and even its most brutal 1-bit quantization is a 46.1GB file (NVIDIA announcement, official NVFP4 repo).

That one number settles the first question most readers will ask. No consumer GPU — not a used RTX 3090, not an RTX 5090 — holds this model. The machines that do are the 128GB unified-memory boxes this site has been comparing all year, which is why the real story here is not “how smart is it” but “which $3,500–$5,100 machine should carry it.”

What NVIDIA actually shipped

The spec sheet, from the Hugging Face repos and NVIDIA’s announcement:

  • 120B total parameters, 12B active. A sparse hybrid-reasoning MoE: each token only reads the activated experts, so decode speed behaves like a ~12B model while the full 120B must sit in memory. Same architecture family as Bolt and Lightning.
  • Multimodal in, text out. Text, images, documents, and video frames as input — NVIDIA pitches it at “long-running AI agents” that read screenshots, PDFs, and dashboards, not just chat.
  • 1M-token context window. More on what that costs you in practice below — the short version is you should not actually launch it at 1M.
  • Two official checkpoints: BF16 for fine-tuning and NVFP4, a 4-bit format that weighs 75.1GB and runs fastest on Blackwell-class hardware. Community GGUFs from Unsloth cover everyone else.
  • OpenMDW-1.1 license. Same genuinely permissive license as Bolt: commercial use, modification, and redistribution without user caps. The Ollama listing carries the same license.

Benchmark claims deserve a flag. MarkTechPost’s launch coverage lists 89.90 on GPQA Diamond and 71.07 on SWE-bench Verified, while other launch-day reports noted NVIDIA hadn’t published a full score table yet — and NVIDIA’s own “up to 5x higher throughput” line refers to its speculative-decoding recipe, not model quality. Treat the scores as vendor-reported until independent evals land. What’s not in dispute: day-0 support shipped across the whole local stack — vLLM, llama.cpp (support merged in PR #30520 on release day, alongside Bolt), Ollama, LM Studio, and Unsloth GGUFs.

The memory math: why this is a 128GB model

Here’s every size that matters, from the Unsloth GGUF repo and the official NVFP4 checkpoint:

QuantizationFile sizeMinimum memory classWhat holds it
1-bit (UD-TQ1)46.1GB~64GBStrix Halo 64GB, 64GB Macs — tight
UD-Q3_K_XL~65GB96GBRTX PRO 6000 96GB, quad-3090 rigs
NVFP4 (official)75.1GB96–128GBDGX Spark (the intended target)
UD-Q4_K_XL (recommended)86.9GB128GBStrix Halo 128GB, Mac M5 Max 128GB, DGX Spark
8-bit132GB192GB+Mac Studio M5 Ultra territory

Unsloth’s own guidance matches the table: roughly 64GB of RAM or unified memory for the 1-bit build, 128GB recommended for 4-bit, and UD-Q4_K_XL named as the best quality-per-GB choice. The Ollama library build lands in the same band at 80–90GB.

Read the table from the bottom of the consumer GPU market and the verdict is immediate:

  • 24GB and 32GB cards (RTX 3090/4090/5090): no. The smallest file is roughly double a 3090’s VRAM. CPU offload at this scale means single-digit tokens per second — the same dead end as running a 70B dense model on 24GB, but heavier.
  • Dual RTX 3090 (48GB): no. 46.1GB of weights into 48GB leaves under 2GB for KV cache and the vision stack. It doesn’t fit in practice.
  • 96GB (RTX PRO 6000, quad-3090): only at 3-bit-class quants, and on the quad-3090 path you’re splitting an MoE with a vision tower across four cards to hold a reduced-quality build. The $13,998+ RTX PRO 6000 technically has room for NVFP4 at modest context, but nobody should buy one for this when a $4,699 Spark is the model’s own design target.
  • 128GB unified memory: yes, comfortably. The 86.9GB Q4 file plus KV cache plus the vision encoder fits with room to breathe. This is the model’s natural home — a point the first community guides made in their titles: you need 128GB, not a Blackwell GPU (DEV Community walkthrough).

Measured speed: the DGX Spark numbers

The only published, measured decode numbers so far come from the DGX Spark, because NVIDIA shipped a Spark-specific vLLM config in the model card — including LiLiCorr, a bundled speculative-decoding draft model.

A first-touch test by Classmethod ran the model card’s recipe as written on a single Spark (NVFP4, 128GB):

  • 19.06 tok/s single-stream decode without the draft model
  • 38.07 tok/s with LiLiCorr speculative decoding — almost exactly 2×
  • Prefill 2,100–2,200 tok/s on 16K–64K inputs; time-to-first-token at 64K input ≈ 30 seconds
  • Startup took 11 minutes at the default 1M context setting

A second data point from the NVIDIA developer forums reports 34–49 tok/s decode with LiLiCorr at contexts up to 16K — and documents the two real problems below.

Sanity-check against the bandwidth ceiling, since decode speed is memory-bandwidth-bound: the NVFP4 checkpoint stores 120B parameters in 75.1GB (~0.63 bytes/param), so each token reads roughly 12B × 0.63 ≈ 7.5GB of expert weights. On the Spark’s 273 GB/s that puts the vanilla ceiling near 36 tok/s; the measured 19 tok/s is ~52% of ceiling, which is typical vLLM-on-Spark efficiency. The 38 tok/s figure doesn’t violate the ceiling — speculative decoding verifies several draft tokens per weight pass, which is exactly how it buys speed.

What that math predicts for the other 128GB machines (estimates)

No one has published Super VL decode numbers for Strix Halo or Apple Silicon yet, so these are estimates from the bandwidth formula plus each platform’s measured efficiency on gpt-oss-120b, the closest well-benchmarked MoE — flag them as such until community numbers land:

MachineBandwidthEstimated Q4 decode
Strix Halo 128GB (EVO-X2, Framework Desktop)256 GB/s~12–18 tok/s (est.)
DGX Spark (NVFP4 + LiLiCorr, measured)273 GB/s19 / 38 tok/s
Mac Studio M5 Max 128GB614 GB/s~25–40 tok/s (est.)

Two asymmetries worth noting. The Mac’s raw bandwidth is 2.4× the Spark’s, but it gets neither the NVFP4 checkpoint nor the LiLiCorr draft — llama.cpp on Metal runs the GGUF, and Unsloth notes the model’s built-in multi-token prediction is on by default there, so the practical gap will be narrower than raw bandwidth suggests. And Strix Halo, the cheapest seat at this table, will likely sit in the low-to-mid teens: usable for agent workflows that read a document and act, slow for long interactive reading if you want more than reading speed (~7–10 tok/s is comfortable reading pace).

Running it: commands and the two real problems

The llama.cpp path works the same on any of these machines (Unsloth guide):

llama-server -hf unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF:UD-Q4_K_XL \
  --ctx-size 16384

Or on Ollama (0.40.5 or later — support landed in the mid-October releases):

ollama run nemotron-3.5-super
# pulls the 80-90GB build; first load takes several minutes from NVMe

Community-recommended sampling from the launch guides: temperature 1.0 / top_p 0.95 for text work, dropped to 0.6 / 0.95 for vision tasks.

Problem 1: the 1M-token default will eat your startup time (and your memory). Classmethod’s 11-minute Spark startup happened because the model card’s vLLM recipe initializes the full 1M context. The fix is the --ctx-size 16384 above (or --max-model-len in vLLM): start at 16K, raise it only when a workload actually needs more. Every GB the KV cache doesn’t reserve is a GB of headroom on a machine that’s already carrying 87GB of weights.

Problem 2: one Spark user reports hard power-offs at 32K+ context. The forum thread describes clean runs up to 16K, then hard shutdowns with no logged error at 32K and beyond — with ~20GB of memory still free, pointing at power/thermals rather than OOM. It’s a single unit’s report, not a confirmed platform defect, but it matches the general pattern that long-context prefill is the heaviest sustained load these small boxes ever see. Until it’s resolved, treat ≤16K as the safe envelope on Spark and watch that thread. The same user’s sweep also shows LiLiCorr’s acceptance length degrading from ~6.5 to ~2.7 tokens per step as context deepens — meaning the 2× speculative speedup is a short-context number, and at depth you should expect decode to drift back toward the vanilla ~19 tok/s.

What to actually buy

Prices as of October 2026, all cross-checked this month:

Your situationThe machinePriceWhere
Cheapest box that runs the recommended Q4 buildGMKtec EVO-X2 128GB~$3,499–$3,649Check price
Want the measured 38 tok/s recipe and the CUDA toolchainNVIDIA DGX Spark$4,699Check price
Highest likely decode speed at 128GB, macOS stackMac Studio M5 Max (128GB config ~$5,099)$2,499 base / ~$5,099 as neededCheck price
Not ready to spend $3,400+ — test the model on rented hardware firstCloud GPU with 96GB+ VRAM, pay per hourhourlyVast.ai

The Framework Desktop 128GB (~$3,449, sold direct by Framework) is the same Strix Halo silicon as the EVO-X2 at essentially the same price — that comparison is here if you’re choosing between them. And if you’re weighing the Spark against the Mac more broadly than this one model, that head-to-head is here.

Before buying anything for a specific model, run your exact model + context combination through the VRAM calculator — Super VL’s vision stack and context setting move the requirement by tens of GB.

Super VL or Bolt 74B?

The October 15 drop was really two answers to different budgets, and NVIDIA was unusually honest about the split:

  • You own a 24GB card → Bolt 74B is the one packaged for you: 22GB native 2-bit GGUF, text-only, agent-focused. Nothing about Super VL changes that verdict.
  • You own (or are buying) a 128GB box → Super VL is now the strongest open vision-language model that fits it — bigger than Qwen and Gemma’s consumer-tier VL options, far below the 256GB floor of Meta’s Muse Spark Open. The 128GB machines finally have a flagship-class multimodal model sized exactly for them.
  • You run coding agents against a local backend → the 71.07 SWE-bench Verified claim, if it holds up independently, puts this in local-coding-flagship territory; pairing it with Cline or other BYOK tools is covered at aicoderscope.com.

For the self-hosted serving stack around it (llama.cpp flags, vLLM configs, multimodal API quirks), the FOSS side lives at aifoss.dev.

FAQ

Can I run Nemotron 3.5 Super VL on an RTX 4090 or 5090? No. The smallest 1-bit GGUF is 46.1GB — bigger than any consumer card’s VRAM. Offloading to system RAM works mechanically but lands in single-digit tok/s. The 24GB-card model from this release is Bolt 74B.

What’s the minimum memory to try it at all? About 64GB of unified memory or system RAM for the 1-bit build, per Unsloth. Quality at 1-bit is a real downgrade; 128GB and the UD-Q4_K_XL build is the configuration the launch guides recommend.

Is the 1M context usable? It exists, but every practical report says start at 16K. Startup at the 1M default took 11 minutes on a Spark, KV cache eats your weight headroom, speculative-decoding gains fade with depth, and one Spark unit hard-crashes past 32K. Long-context is this model’s marketing, not yet its strength on $4,000 hardware.

Does it replace Qwen3.6/Qwen3.8 on my 24GB card? No — it doesn’t fit there. On 128GB machines it competes with the larger open MoEs; its unique sell is vision + documents + video input under a truly permissive license.

Is the license actually open for commercial use? Yes. OpenMDW-1.1 grants use, modification, and redistribution without user caps or revenue triggers — more permissive than Meta’s Muse Model License with its $20M revenue clause.

Sources

Last updated October 19, 2026. Prices and specs change; verify current rates before purchasing.

Products linked in this article, all picked for the 128GB tier this model actually needs:

Was this article helpful?

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.