Speculative Decoding for Local LLMs in 2026: The Setup That Doubles Your Tokens per Second (and When It Backfires)

By RunAIHome Team · 12 min read
llama-cppspeculative-decodinglocal-llmollamaperformance

TL;DR: Speculative decoding pairs a small “draft” model with your big one to guess several tokens per step, and it can cut your generation time by 1.5–3× with identical output. In 2026 the flags were renamed and Qwen3.6/Gemma 4 ship built-in MTP drafters, so the setup is easier than ever — but on a tight GPU a draft model can make you slower, not faster.

What you’ll be able to do after this guide:

  • Turn on speculative decoding in llama.cpp, Ollama, or LM Studio with the correct 2026 flags (the old --draft-max command now errors out).
  • Use the new MTP path for Qwen3.6 and Gemma 4 — no separate draft model download for most models.
  • Tell in 60 seconds whether speculative decoding is actually helping you or quietly costing you tokens/second.

Honest take: If your model already runs 100% on the GPU and you generate mostly predictable text (code, JSON, boilerplate), speculative decoding is close to free speed. If you’re VRAM-constrained or writing high-entropy creative prose, it can backfire — verify with a benchmark before you commit VRAM to a draft model.

What speculative decoding actually does (30-second version)

Text generation is memory-bandwidth-bound: each new token requires reading the entire model’s weights from VRAM. A 27B model at Q4 reads ~16 GB per token, which is why a fast card like a used RTX 3090 (936 GB/s) tops out around 95 tokens/sec on a 7B and far less on a 27B.

Speculative decoding attacks that wall with a trick: a small, cheap draft model guesses the next N tokens in a fraction of the time, then your large target model verifies all N of them in a single forward pass. Every guess the target agrees with is a token you got almost for free. Reject a guess and you fall back to normal decoding for that token — so the output is bit-for-bit identical to running the big model alone. It is a pure latency optimization, not a quality trade-off.

The catch is the acceptance rate. When the draft’s guesses are accepted 70%+ of the time, you win big. When they’re mostly wrong, you pay for the draft model’s compute and the wasted verification, and you can end up slower than baseline. We cover exactly when that happens below. For the deeper theory, see our companion piece on why local LLMs got good in 2026.

The #1 gotcha: your old command now errors out

If you copy a 2024- or 2025-era speculative-decoding command into a current llama.cpp build, it fails with an “unrecognized argument” error. That’s because the legacy flags were removed and renamed under a --spec- prefix in the 2026 reworking of the CLI. This is the single most common reason a working command from an old blog post suddenly breaks.

Old flag (removed)New flag (2026)Meaning
--draft-max--spec-draft-n-max NMax draft tokens per step (default: 3)
--draft-min--spec-draft-n-min NMinimum draft tokens per step
--draft-p-min--spec-draft-p-min PMin draft probability to keep speculating (default: 0.00)
-ngld-ngld / --spec-draft-ngl NDraft-model layers to offload to VRAM
(new)--spec-type [draft|draft-mtp|draft-eagle3|ngram-simple]Which speculative method to use

The draft-model flag itself, -md, --model-draft FNAME, is unchanged. If your command “used to work,” swap --draft-max→--spec-draft-n-max and --draft-min→--spec-draft-n-min first — that fixes the majority of breakages.

Method 1: A classic draft model in llama.cpp

This is the universal approach: run a small model from the same family as a drafter. Same family matters because the two models must share a tokenizer/vocabulary — a Llama drafter for a Llama target, a Qwen drafter for a Qwen target. Mismatched vocab either refuses to load or tanks your acceptance rate.

A working llama-server invocation pairing an 8B target with a 1B drafter:

./llama-server \
  -m   ./models/Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -md  ./models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
  -ngl 99 -ngld 99 -fa on \
  -c 8192 \
  --spec-draft-n-max 8 --spec-draft-n-min 1 \
  --port 8080
  • -md points at the draft GGUF; -ngld 99 puts all its layers on the GPU (a draft model that spills to CPU defeats the purpose).
  • -fa on enables Flash Attention — important, because speculative decoding leans on batched verification.
  • --spec-draft-n-max 8 drafts up to 8 tokens per step. Higher is better for predictable output like code, worse for creative text where guesses get rejected. Start at 4–8.

DataCamp’s measured pairing of Llama 3.1 8B (target) with Llama 3.2 1B (draft) reported a 1.83× throughput gain at a draft length of 5. Community results across well-matched pairs land in the 1.5–3× range, with code and structured output at the top of that band. On a card like the RTX 4090, an 8B that runs ~130 tok/s alone has been reported to push past 180 tok/s with a 1B drafter attached.

The cost you pay: the draft model needs its own VRAM. A 1B at Q4 is ~1 GB, which is trivial on 24 GB but can be the difference between “fits” and “spills to CPU” on an 8–12 GB card. If adding the drafter forces your target to offload layers, you’ll lose more than you gain. (If you’re already fighting for memory, our CUDA out of memory fixes apply here too.)

Method 2: MTP — the 2026 shortcut for Qwen3.6 and Gemma 4

The bigger 2026 change is Multi-Token Prediction (MTP): newer models are trained with extra prediction heads that draft the next few tokens themselves, so you don’t need a separate draft model at all. llama.cpp exposes this through --spec-type draft-mtp.

Alibaba trained MTP directly into Qwen3.5 and Qwen3.6. For most of them the heads are baked in; Qwen3.6 is the one that still needs a separate MTP GGUF alongside the main weights. A minimal command:

./llama-server \
  -m Qwen3.6-27B-Q4_K_S.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -c 8192 -fa on \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0

Reported results:

  • Qwen3.6-27B dense on an NVIDIA A10G (24 GB): ~25 → ~45 tok/s (+78%) with --spec-type draft-mtp --spec-draft-n-max 2 (Victor Mustar / Hugging Face).
  • Qwen3.6-27B on Apple Silicon: ~7 → ~16 tok/s at an 82% acceptance rate with --spec-draft-n-max 2 (community gist), i.e. roughly a 2× win where the model was previously borderline unusable.

Note --spec-draft-n-max 2 — MTP drafters are typically tuned for a smaller draft length than a standalone model, because the built-in heads are most confident about the very next tokens.

Ollama: the DRAFT Modelfile directive

Ollama added MTP speculative decoding on the MLX runner first (Apple Silicon), gated behind an --experimental flag when it landed. It introduced a new DRAFT directive in the Modelfile:

FROM /path/to/gemma4/main
DRAFT /path/to/gemma4/draft

You can also build a draft-enabled model directly:

ollama create --experimental -f Modelfile mymodel \
  --quantize nvfp4 --draft-quantize nvfp4

Then call it through the normal OpenAI-compatible endpoint — nothing else changes. A later Ollama release turned the Gemma 4 MLX path on by default, with one benchmark run reporting Gemma 4 on MLX ~90% faster on a coding-agent workload at identical output. Because Ollama’s version numbering and runner support have moved fast in 2026, check ollama --version and the release notes before assuming MTP is wired up on your platform — it reached the MLX runner well before other backends. For the broader MLX story, see our Ollama v0.30 MLX guide.

LM Studio: the one-toggle path

If you’d rather not touch a command line, LM Studio shipped speculative decoding as a GUI feature (v0.3.10). You load your main model, then pick a compatible draft model from a dropdown in the model settings — LM Studio filters the list to models that share the target’s vocabulary, which saves you from the mismatched-tokenizer trap. It’s the least flexible but most foolproof route, and a good way to A/B test whether a draft model helps your workload before scripting it in llama.cpp.

Real numbers: what speedup to actually expect

SetupHardwareBaselineWith spec. decodingSource
Qwen3.6-27B + MTP (n-max 2)A10G 24 GB25 tok/s45 tok/s (+78%)HF / V. Mustar
Qwen3.6-27B + MTP (n-max 2)Apple Silicon~7 tok/s~16 tok/s (82% accept)Community gist
Llama 3.1 8B + Llama 3.2 1B draftGPU, draft len 51.0×1.83×DataCamp
Gemma 4 + MTP (coding agent)Apple Silicon (MLX)1.0×~1.9× (+90%)Ollama release
Gemma 4 + int4 draft (bad match)Apple Silicon60 tok/s45 tok/s (slower)Ollama PR #15980

That last row is the point of the next section.

When speculative decoding backfires

Speculative decoding is not free, and the honest failure modes are:

  1. Low acceptance rate = net slowdown. A tester running Gemma 4 with an int4 draft model saw only a 41% token match rate, which dropped generation from 60 tok/s to 45 tok/s — the draft was wrong so often that the wasted work outweighed the wins. Below roughly 60–70% acceptance you’re usually losing.
  2. VRAM pressure. The draft model occupies memory. On a 24 GB card that’s a rounding error, but on 8–12 GB it can push your target model into CPU offload, and partial-CPU inference is dramatically slower than any speculative gain. If you’re already near the edge, don’t add a drafter — see our notes on running big models on 24 GB.
  3. Creative/high-entropy text. Speculation shines on predictable output — code, structured data, repetitive formatting — where the draft guesses right. Open-ended creative writing has too many equally-likely next tokens, so acceptance craters and the speedup with it.
  4. Concurrent serving. The gains are a single-stream, latency-focused optimization. If you’re batching many requests (a busy vLLM-style server), you’re already throughput-bound and speculative decoding helps far less. If that’s your workload, our vLLM vs Ollama comparison is the better starting point.

The two-minute test: run the same prompt with and without the draft model and watch the tok/s in the server log. If it didn’t clearly go up, drop the drafter and reclaim the VRAM. Quantization also matters here — a badly-matched draft quant is a common cause of low acceptance, and our Q4 vs Q5 vs Q6 vs Q8 breakdown explains the trade-offs.

Tuning checklist

  • Match the family and vocabulary. Qwen draft for Qwen, Llama for Llama. Mismatched tokenizers kill acceptance.
  • Keep the draft ~10× smaller than the target. A 1B drafting for an 8B–14B is the classic ratio; MTP heads are effectively even cheaper.
  • --spec-draft-n-max: start at 8 for a standalone draft on code, 2–4 for MTP or creative text. More drafted tokens only pay off at high acceptance.
  • -ngld 99: always keep the draft model fully on the GPU. A CPU-resident drafter is pointless.
  • -fa on: enable Flash Attention; batched verification depends on it.
  • Still slow? Confirm the target is 100% on GPU first — no amount of drafting fixes CPU spillover. Our Ollama speed guide covers that baseline.

No GPU with room for both models? This is a good case for renting a 24–48 GB card by the hour on RunPod to benchmark the pairing before you buy hardware. And if your real goal is a faster coding loop, the model and tooling choice matters as much as the decoder — see aicoderscope.com for local-first coding setups and aifoss.dev for the open-source stack.

FAQ

Does speculative decoding change the output quality? No. It’s mathematically lossless — the target model verifies every token, so the result is identical to running the big model alone. Only speed changes.

Do I need to download a separate draft model? For the classic method, yes — a small same-family model. For MTP-trained models (Qwen3.5/3.6, Gemma 4), the drafter is built in; Qwen3.6 is the notable one that still ships a separate MTP GGUF.

Why did my old --draft-max command stop working? The flag was renamed to --spec-draft-n-max in the 2026 llama.cpp CLI rework. Use the new --spec- names (see the table above).

Will it help on my 8 GB card? Maybe not. The draft model needs VRAM, and if it forces your target into CPU offload you’ll be slower overall. Benchmark it; if tok/s doesn’t rise, remove the drafter.

What acceptance rate do I need? Roughly 60–70%+ to come out ahead. Below that, the wasted verification usually cancels the gains — as in the 41%-match case that dropped from 60 to 45 tok/s.

MTP vs a separate draft model — which is faster? MTP is lower-overhead (no second full model to load) and is the easiest win on supported models. A well-matched standalone draft can hit higher speedups on very predictable output, at the cost of extra VRAM.

Sources

Last updated July 4, 2026. Tokens/sec figures are hardware- and quant-dependent; benchmark your own pairing. Flag names and runner support change between builds — verify against your installed llama.cpp / Ollama / LM Studio version.

  • RTX 3090 — 24 GB / 936 GB/s; enough headroom to host a target model plus a draft model comfortably.
  • RTX 4090 — highest single-card tok/s for local inference, where speculative decoding pushes an 8B past 180 tok/s.

Was this article helpful?