NVIDIA Nemotron 3.5 Bolt E4B for Local AI: The "4GB" Agent Model Is Really a 6.18GB File — What Runs It in 2026

By RunAIHome Team · 12 min read
nemotronnvidialocal-llmgemma-4edge-aijetsonrtx-3060ollamallama-cppagentic-ai

TL;DR: NVIDIA’s Nemotron 3.5 Bolt E4B landed on Hugging Face October 23, 2026 — a Nemotron post-train of Gemma 4 E4B (8B total, ~4B effective parameters) under the permissive OpenMDW-1.1 license, built for low-latency agents on small hardware. The “4GB” in the name is runtime VRAM, not file size: the only official GGUF is 6.18GB. If you own any 8GB GPU, it costs you nothing to run.

Nemotron 3.5 Bolt E4BGemma 4 E4B QATQwen 3.5 4B
Best forTool-calling agents that must obey the system promptProse, chat, writingRaw function-calling accuracy
File that fits small cards6.18GB Q4_K_M (official)~5GB QAT2.74GB Q4_K_M
Standout scoreIHEval 84.79 (next best: 67.04)EQ-Bench v3 41.17BFCL-v3 70.95
The catchBigger file than its own parent modelWeaker at knowing when not to call a toolFalls behind on multi-turn agent tasks

Honest take: if you’re building a local agent that has to follow instructions under pressure, Bolt E4B is now the default at this size — its instruction-hierarchy score isn’t just better, it’s 17 points clear of everything in its class. For everything else (chat, writing, general Q&A), the Gemma 4 E4B you may already have pulled is the same speed and slightly nicer to talk to. Nobody should buy a GPU for this model — that’s the point of it.

NVIDIA promised three Nemotron 3.5 models in October: Super VL 120B and Bolt 74B shipped October 15, and the small one — Bolt E4B, slated for October 23 — arrived on schedule. We held this article until the weights were confirmed live on NVIDIA’s official Hugging Face org, because the announcement coverage left basics like license and parameter count undisclosed right up to release day. They’re disclosed now, and some of them are not what the marketing name suggests.

What actually shipped on October 23

The facts, from the official repo and NVIDIA’s model page:

  • Architecture: a post-trained version of Gemma 4 E4B-it — Google’s MatFormer design with 8B total parameters and roughly 4B effective per token. NVIDIA did the Nemotron agent/instruction post-training on top; the skeleton is Gemma.
  • License: OpenMDW-1.1, same as Bolt 74B and Super VL. Commercial use and modification permitted.
  • Context: 64K tokens in, 64K out. Text-only in both directions — community inspection found Gemma 4’s vision encoder tensors still present in the BF16 checkpoint, but NVIDIA doesn’t document or evaluate vision for this model, so treat it as unsupported.
  • Files: exactly two precisions — a 15.88GB BF16 checkpoint and a 6.18GB Q4_K_M GGUF, plus a DFlash speculative-decoding drafter file. There is no 4GB file in the repository.
  • Reasoning switch: thinking mode is controlled per-request via the chat template (enable_thinking), with a runtime thinking budget. NVIDIA recommends temperature 1.0 and top_p 0.95 in both modes.

The “4GB” name refers to NVIDIA’s stated runtime VRAM figures — 3.1 to 4.3GB measured — which the model hits because Gemma’s per-layer embeddings don’t have to sit in GPU memory. On disk, and over a 6GB card’s memory bus, it behaves like the ~6GB model it is. Worth knowing before you point a 4GB card at it expecting a comfortable fit: the weights alone are tight there, and KV cache for long context comes on top. Our VRAM calculator will tell you where your exact card lands.

One mildly awkward detail: the official Q4_K_M is larger than the ~5GB QAT build of its own parent model, Gemma 4 E4B. Google trained the quantization loss out of its QAT checkpoints; NVIDIA shipped a standard post-training quant of a checkpoint that still carries the (unused) vision tensors. On a 6GB card, the parent fits more comfortably than the child.

The benchmark story: one huge number, several small ones

All scores below are NVIDIA-reported BF16 numbers from the model card — no independent benchmark suite has run yet, so treat them as vendor claims with that grain of salt.

Benchmark (what it measures)Bolt E4BGemma 4 E4B-itQwen 3.5 4B
IHEval (obeying instruction hierarchy)84.7967.0460.57
Tau2-Bench (multi-turn customer tasks)55.138.5—
MultiChallenge (multi-turn robustness)53.1042.0851.76
ToolSandbox (tool use in context)68.7765.43—
When2Call MT (knowing when NOT to call)63.3662.90—
BFCL-v3 (function calling)68.5166.3970.95
EQ-Bench v3 (emotional intelligence/prose)39.4041.17—

The pattern is coherent: NVIDIA’s post-training bought agent behavior, and the price was a sliver of Gemma’s conversational polish. The IHEval number is the one that matters — 17.75 points clear of the nearest 4B-class competitor is an unusual margin at any size, and instruction hierarchy (does the model keep following the system prompt when the user, or a document it just read, tells it to do something else?) is exactly what breaks small local agents in practice. Meanwhile Qwen 3.5 4B still wins raw single-shot function calling, so if your workload is “turn text into one API call,” nothing changed.

The only independent test published so far supports the agent framing from a different angle. Atomic Agent ran Bolt E4B and Gemma 4 E4B head-to-head as a local agent on an RTX 5090, 18 runs each: Bolt passed 12/18 vs Gemma’s 11/18 — a one-run difference the author correctly flags as noise at that sample size. The non-noise finding: Bolt generated roughly a tenth of the tokens to finish the same tasks, so despite a slightly lower raw speed (178 vs 193 tok/s on that card), it finished far sooner in wall-clock time. For an edge agent, tokens-not-written is the cheapest speedup there is.

Measured speed, and what your card will do

Three measured data points exist as of October 24, all llama.cpp, batch 1:

HardwareDecode speedTime to first token
Jetson Orin Nano 8GB13.0–15.5 tok/s5.8–6.3s
RTX 5070 12GB106.2–133.4 tok/s0.55–1.24s
RTX 5090 32GB178 tok/s—

Two useful things fall out of these numbers.

First, the RTX 5070 result sits essentially at the card’s memory-bandwidth ceiling (672 GB/s ÷ ~5GB of weights read per token ≈ 134 tok/s), which means decode is bandwidth-bound on midrange cards. Using that implied ~5GB/token traffic, the ceilings for cards we haven’t seen measured — flagged as computed, not measured: a used RTX 3060 12GB (360 GB/s) tops out around 72 tok/s, so expect roughly 50–68 tok/s in practice; an RTX 4060 8GB (272 GB/s) around 38–52 tok/s. Both are far past the ~7–10 tok/s reading-speed bar — any dedicated GPU from the last five years runs this model comfortably.

Second, big cards are wasted on it. The RTX 5090 has 2.7× the 5070’s bandwidth but delivers only ~1.3–1.7× the speed — at 4B effective parameters, kernel-launch overhead starts dominating before the memory bus does. If you were looking for a reason to buy a flagship card, this model is arguing the opposite.

The Jetson numbers deserve their own honest sentence. 13–15 tok/s is fine for an embedded agent chewing through tool calls in the background; a nearly six-second time-to-first-token is not a real-time voice assistant, whatever the “conversational AI at the edge” framing says. If the TTFT matters for your use case, the fix is a discrete GPU, not a bigger Jetson.

How to run it today

The base architecture is Gemma 4, which llama.cpp and Ollama have supported since June — unlike Bolt 74B, which needed a same-day llama.cpp PR, E4B runs on whatever recent build you already have. The official repo’s quick-start downloads the GGUF directly:

$ hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB --include "*Q4_K_M*"
$ llama-server -m NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-Q4_K_M.gguf -c 16384 -fa on

Ollama users can pull the GGUF straight from Hugging Face — there’s no tag in the Ollama library yet:

$ ollama run hf.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF:Q4_K_M
>>> /set parameter temperature 1.0

Verify GPU residency with ollama ps afterward — the PROCESSOR column should read 100% GPU (if it says CPU, start here).

A real gotcha from the first 24 hours: the thinking-mode default depends on who’s serving. The chat template itself defaults enable_thinking to false, but llama-server’s --reasoning auto behavior switches thinking on for this model, per Tere Labs’ GGUF notes — so the same prompt can produce a 40-token answer in one stack and a 400-token reasoning chain in another, and the first symptom is “why is my edge agent suddenly 10× slower?” The fix: set it explicitly. Pass "chat_template_kwargs": {"enable_thinking": false} in your OpenAI-compatible request (or true, if you want the reasoning), and don’t trust either default. Given that token frugality is this model’s whole wall-clock advantage, leaving thinking accidentally enabled throws away the thing you picked it for.

Unsloth published GGUFs and a run guide same-day and states the model runs locally in 6GB of RAM; unofficial MLX conversions (UraionLabs) already exist for Mac users, but they’re community builds — same caveat we gave Bolt 74B’s early MLX ports.

What to actually buy

Prices as of October 2026. The honest headline: most readers should buy nothing — that’s what a 4GB-class model is for.

Your situationThe machinePriceWhere
Any GPU with 6GB+ VRAM already in your machineNothing — pull the GGUF tonight$0—
No discrete GPU; want the cheapest card that runs E4B and 12B-class modelsUsed RTX 3060 12GB~$230–$300 (eBay asks)Check price
Building an always-on, low-power embedded agent boxJetson Orin Nano Super dev kit$249 list — stock is spotty, expect markupsCheck price
Want E4B now and the 24GB-tier Bolt 74B laterUsed RTX 3090 24GB~$1,456–$1,558Check price
Want to test the bigger Nemotrons before buying anythingRented 3090, from ~$0.07/hrpay per hourVast.ai

One caution on the budget row: don’t confuse used prices with the new-retail situation. NVIDIA’s RTX 3060 relaunch has new-retail listings up roughly 45% in two months and approaching $500 per Guru3D — at that price the card makes no sense. The $230–$300 used market is the one that does.

FAQ

Does Bolt E4B fit a 4GB card? Marginally, and not comfortably. NVIDIA’s measured VRAM floor is 3.1GB with embeddings held off-GPU, which leaves almost nothing for KV cache. On 4GB, the parent Gemma 4 E2B QAT (~3GB, our guide) is the saner pick. From 6GB up, E4B is fine.

Should I replace Gemma 4 E4B with it? Only if you’re running agents. For tool calling under a system prompt, yes — the IHEval/Tau2 gaps are large. For chat and writing, Gemma’s QAT build is smaller on disk and scores slightly higher on prose quality; there’s no speed difference that matters.

Is it multimodal like Gemma 4 E4B? No. Inputs and outputs are text-only per the model card. The vision tensors are physically present in the BF16 checkpoint but undocumented and unevaluated — don’t build on them.

Can it drive a coding agent? It can drive the agent loop — that’s what it’s tuned for — but a 4B-effective model won’t write the code a 27B will. The realistic edge pattern is E4B as the router/tool-caller with a bigger model behind it. For wiring a local model into Cline or Continue, see our sister site’s local coding stack setup.

Where does it sit in the Nemotron 3.5 family? Smallest of three: E4B (this one, any GPU), Bolt 74B-A9B (24GB cards), Super VL 120B-A12B (128GB-class machines). Same OpenMDW-1.1 license across the family. For the server-side setup angle, aifoss.dev’s Ollama review covers the self-hosting stack.

Products linked above, for the situations where buying makes sense:

Sources

Last updated October 24, 2026. Prices and specs change; verify current rates before purchasing.

Was this article helpful?

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.