Oct 25, 2026

llama.cpp Decision Models at Home in 2026: The Real VRAM Math for Running a Router Beside Your 27B Main Model

Q: Does a decision router speed up my main model?

No. It doesn't touch decode speed — that's still [bandwidth ÷ bytes-per-token](/blog/qwen38-27b-consumer-gpu-hardware-guide-2026/). What it saves is *which* model runs: sending "fix this typo" to a 4B instead of a 27B returns the answer several times sooner and frees the big model. Don't confuse it with speculative decoding, which is a different technique with its own draft-model rules.

By RunAIHome Team · 14 min read

llama-cppdecision-modelslocal-llmgpuhardware-guideroutingagents

TL;DR: llama.cpp merged a /v1/systemone endpoint on October 2, 2026 (PR #29818) that serves decision models — models that score options in one forward pass instead of generating text. Five open GGUFs shipped with it, from 144M to 27B. The two smallest run in under 1GB beside anything you already own; only the 4B-class routers next to a 27B main model force a 24GB card.

	Julia-1 / Laya	Kev-4B / lev	OpenJev
Best for	High-volume routing and filtering on any hardware, after fine-tuning	Zero-shot routing, tool selection, agent output checks	Vision-capable judging, hardest decision tasks
Size (params / disk)	144M / 421M, under 1GB	4B; Kev-4B Q4_K_M is 3.03GB	27B; Q4_K_M is 16.5GB
VRAM impact	Negligible — runs even on CPU at ~63ms warm	~4GB — fits beside a 12B main on 16GB, beside a 27B main on 24GB	Needs a 24GB card to itself
The catch	Weak zero-shot (0.36 accuracy vs a 0.46 majority-class baseline) — must fine-tune	Q4_K_M agrees with BF16 only 93.2% of the time; use Q8_0 (98.4%)	CC BY-NC 4.0 — non-commercial only

Honest take: You almost certainly don’t need new hardware for this. Laya routes in 8–10ms on a GPU and ~63ms warm on a bare CPU, and Kev-4B’s Q8_0 is a 4.48GB file that fits in the VRAM most setups already have left over. The only configuration that genuinely forces an upgrade is holding a 4B router and a 27B main model resident at once — that’s a ~21GB stack, which means a used RTX 3090 24GB ($1,456–$1,558, October 2026).

Check your own model-plus-context combination in our VRAM calculator before taking any table below at face value.

What actually shipped on October 2

PR #29818 (“llama, server: add /v1/systemone API”) was opened October 1, 2026 and merged a day later. It adds a decision head to llama.cpp’s graph and a new server endpoint, /v1/systemone, that speaks the System One request format introduced by TypeSafe’s Jev — the hosted decision model that charges $0.042 per million input tokens with free output (there is no text output to bill). Existing Jev clients only need to change the base URL.

A decision model doesn’t generate text. You send it a state — plain text or JSON describing a situation — plus one or more typed questions, and it returns a probability for each option in a single forward pass. Three question types are supported:

choice — pick one option from a list of labeled criteria (this is the routing case)
noul — a yes/no question, answered with a confidence between 0 and 1
score — a rating against an ordered scale

No sampling, no token-by-token decode, no streaming (the endpoint doesn’t support it, and doesn’t need to). For a home lab, that has a concrete consequence: decision latency is one weight-read plus overhead, not a decode loop. A 3GB router on a 936 GB/s card reads its weights in ~3.2ms. The decode-speed anxieties that dominate every other hardware decision on this site mostly don’t apply here.

The PR author describes decision models bluntly as “fancy wrappers around traditional embedding models (BERT/Qwen/etc)” — which is accurate, and explains the size range below. llama.cpp’s implementation was validated against the reference implementations: worst-case probability difference across the five launch models ranges from 8.4e-4 (Laya) to 2.2e-2 (Julia-1).

The five launch models, sized honestly

All five ship as GGUFs under the ggml-org Hugging Face organization and load with a one-liner like llama-server -hf ggml-org/Kev-4B-GGUF:Q4_K_M.

Model	Params	Base	Disk size	License
Julia-1	144M	mmBERT-small encoder	well under 1GB	Apache 2.0
Laya	421M	ModernBERT-large encoder	under 1GB RAM in independent tests	Apache 2.0
Kev-4B	4B	Qwen3.5-4B-Base + LoRA + pointer head	Q4_K_M 3.03GB / Q8_0 4.48GB / BF16 8.43GB	Apache 2.0
lev	~4B	Qwen3.5-4B	Kev-class (no official size listing yet)	Apache 2.0
OpenJev	27B	— (vision-capable; image input still a TODO in llama.cpp)	Q4_K_M 16.5GB	CC BY-NC 4.0

Three things in that table matter more than the parameter counts:

The quantization-agreement numbers are published, and they’re not 100%. TurinTech’s comparison against the BF16 original found Kev-4B’s Q4_K_M gives the same decision 93.2% of the time over 2,000 test decisions; Q8_0 agrees 98.4%. For a chat model, a quant that “mostly agrees” is fine — the text still reads well. For a router, a 6.8% decision-flip rate is a real error rate stacked on top of the model’s own. The Q8_0 is only 1.45GB bigger. For decision workloads, skip Q4 and take the Q8_0.

OpenJev is non-commercial. The other four are Apache 2.0, but OpenJev’s CC BY-NC 4.0 weights rule it out of anything that touches revenue — including, arguably, a homelab stack that powers paid client work. It’s also 16.5GB at Q4_K_M, which makes it a main-model-sized judge, not a co-resident router. It fits the same 24GB cards a 27B chat model does, and you’d run it instead of something, not beside it.

The BF16 footprint is bigger than the file. Kev-4B’s 8.43GB BF16 build was measured needing 13.1 GiB of GPU memory loaded — buffers and graph overhead on top of weights. The same ratio applies to the quants: budget the file size plus roughly 20–30%, same as any llama.cpp model.

What a decision costs: measured latency vs the Jev API

The queue of community benchmarks that followed the launch gives a consistent picture across three independent tests:

Setup	Measured latency
Laya fp16, RTX 4090, 5-question HTTP request	8.1–9.6ms median (vendor benchmark)
Laya local, under 1GB RAM	~33ms
Laya, M5 Pro GPU	53ms
Laya, bare CPU, warm	63ms (but ~20s cold start)
Jev cloud API (same tests, network included)	236–276ms

The pattern worth noticing: on the sub-1B encoder models, the GPU barely matters. A 421M fp16 model reads ~0.84GB of weights per forward pass — on an RTX 4090’s 1,008 GB/s that’s under a millisecond of memory traffic, so the 8–10ms measured is almost entirely tokenization and HTTP overhead. The jump from an RTX 4090 to an M5 Pro to a plain CPU moves the number from ~9ms to ~53ms to ~63ms. Every one of those beats the hosted Jev API by 4–30×, because 200ms+ of the cloud number is network and queueing, not inference.

For Kev-4B on site-canon hardware, the bandwidth ceiling method (the same one we use for every decode estimate) says: 4.48GB Q8_0 weight read ÷ 936 GB/s on an RTX 3090 ≈ 4.8ms theoretical floor per decision, so expect real-world totals in the 10–25ms range once overhead is added. Nobody has published an independent Kev-4B millisecond benchmark yet; treat that as an estimate, not a measurement.

So the latency verdict the queue asked for is unambiguous: local decision models beat the cloud decision API on latency on any hardware you own, including no GPU at all. The honest caveat is that latency was never the expensive part — at $0.042 per million input tokens with free output, Jev costs roughly 4 cents per million tokens of routing. You self-host this for privacy, for the 20,000-decisions-a-day agent loop, or because your requests shouldn’t leave the LAN — the same reasons as everything else on this site — not to save meaningful money at hobby volume.

The VRAM math for a router + main model stack

Here’s the part that actually touches your wallet. llama-server serves one model per process, so a routed stack is two instances on two ports, and their VRAM budgets add. Using site-canon GGUF sizes:

Your card	Main model that fits	Router that fits beside it	Total weights
8GB (RTX 4060)	Gemma 4 E4B QAT, ~5GB	Julia-1 or Laya only	~6GB
12GB (RTX 3060)	Gemma 4 12B QAT, ~7GB	Kev-4B Q8_0, 4.48GB	~11.5GB — tight, Q4 router safer
16GB (RTX 5060 Ti)	Gemma 4 12B QAT, ~7GB	Kev-4B Q8_0 + room for context	~11.5GB
16GB	Gemma 4 26B-A4B QAT, ~15GB	encoder routers only — Kev-4B does not fit	~16GB
24GB (RTX 3090/4090)	Qwen3.8-27B Q4_K_M, 16.8GB	Kev-4B Q8_0, 4.48GB	~21.3GB

Two rows deserve commentary.

The 16GB rows are the trap the headlines skip. A 16GB card runs a 27B-class QAT MoE or a 4B router comfortably — not both. If your main model already fills the card (26B-A4B at ~15GB, or Qwen3.8-27B offloaded), the only routers you can co-locate are the sub-1B encoders, and those need fine-tuning to be trustworthy (next section).

The 24GB row is the configuration the whole topic implies: Qwen3.8-27B Q4_K_M (16.8GB, ~41 tok/s decode on an RTX 3090) plus Kev-4B Q8_0 (4.48GB) is ~21.3GB of weights, leaving under 3GB for both models’ KV caches. It works on a 24GB card with the main model held to ~8–16K context — run nvidia-smi and watch headroom before you commit to long-context work. A practical alternative on any card: run the main model on GPU and the router on CPU. A warm encoder router answers in ~63ms on CPU, which is invisible next to a multi-second generation, and costs zero VRAM.

Running it: the commands

Two terminals, two ports:

# Terminal 1 — main model
llama-server -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_M --port 8080

# Terminal 2 — router
llama-server -hf ggml-org/Kev-4B-GGUF:Q8_0 --port 8081

Then a routing decision is one POST (shape from PR #29818’s test suite; probabilities illustrative — yours will differ):

curl http://localhost:8081/v1/systemone -H "Content-Type: application/json" -d '{
  "state": "User request: Rename the variable `usr` to `user` across three files.",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which model should handle this request?",
      "criteria": {
        "small_fast_model": "trivial edit, no reasoning needed",
        "large_model": "multi-step reasoning or architecture work"
      }
    }
  }
}'
# → {"route": {"small_fast_model": 0.94, "large_model": 0.06}}  (abridged)

Your client reads the probability and sends the actual request to port 8080 or to whatever serves your small model. Total added latency: one forward pass — the 10–60ms from the table above, against cloud-API round-trips of 236–276ms for the equivalent hosted call.

The traps (all three are real, two have fixes)

Pointing /v1/systemone at a normal model fails. The endpoint switches on {arch}.decision.type GGUF metadata, and a GGUF without .decision. metadata is rejected. You cannot route with the Qwen or Gemma GGUF you already have — only the five launch models (and future conversions) work. Fix: it’s by design; pull one of the ggml-org/*-GGUF decision repos. Related: LM Studio will load OpenJev and even chat with it as an ordinary model, but exposes no /v1/systemone on its local server — one early hands-on tester lost an evening to that. Use llama.cpp’s own llama-server (build from October 2, 2026 or later).

Zero-shot encoder routing is worse than guessing the majority class. The most-cited independent review measured Laya at 0.36 zero-shot accuracy on a task where always answering the most common label scores 0.46, and 0.425 on Banking77’s 77-label set where the hosted Jev scores 0.870. The encoder models are fast bases you fine-tune on your own routing data, not drop-in judges. Fix: either collect a few hundred labeled routing examples and fine-tune (Laya’s vendor and third parties both document this), or use Kev-4B/lev, which are tuned for zero-shot use — that’s precisely what their extra 3.5B parameters buy.

CPU cold starts are brutal. The same review measured ~63ms warm CPU inference but ~20-second cold starts. If you run the router on CPU to save VRAM, keep the process resident — don’t spawn it per request. (Same class of problem as Ollama’s model reloading, different layer.)

What to actually buy

Prices as of October 2026, all verified in the comparison above:

Your situation	The move	Price	Where
Main model is ≤12B-class, any card 12GB+	Nothing — Kev-4B Q8_0 fits your leftover VRAM	$0	—
Main model fills a 16GB card	Nothing — run the router warm on CPU (~63ms)	$0	—
Want 27B main + 4B router resident, GPU-fast	Used RTX 3090 24GB	$1,456–$1,558	Check price
Building a first 16GB box around a 12B + router stack	RTX 5060 Ti 16GB	$679–$805 (MSRP $429)	Check price
Want to test the routed stack before buying anything	Rented 3090, from $0.07/hr	pay per hour	Vast.ai

The first two rows are the honest ones: for most existing setups this feature costs nothing. The upgrade case is specifically the person who wants the 27B-plus-router stack resident on one card at GPU latency — and a used RTX 3090 remains the cheapest 24GB that does it, DRAM crisis pricing and all. If you’re starting from zero, start with the GPU buying guide instead of buying for this one feature.

For wiring the routed stack into an actual coding agent as a BYOK backend, our sister site covers the client side in its Continue.dev + Ollama local stack guide, and aifoss.dev’s Ollama review covers the broader self-hosted serving stack (note: Ollama itself has no /v1/systemone yet — this is llama.cpp-only as of late October 2026).

FAQ

Does a decision router speed up my main model? No. It doesn’t touch decode speed — that’s still bandwidth ÷ bytes-per-token. What it saves is which model runs: sending “fix this typo” to a 4B instead of a 27B returns the answer several times sooner and frees the big model. Don’t confuse it with speculative decoding, which is a different technique with its own draft-model rules.

Can I use the router and main model in one llama-server process? No — one model per process. Run two instances on different ports; their VRAM adds.

Is Kev-4B good enough zero-shot, or do I have to fine-tune like Laya? Kev-4B and lev are the zero-shot-usable tier — Kev-4B is explicitly positioned for routing, classification, and policy checks out of the box. The sub-1B encoders (Julia-1, Laya) measurably are not: 0.36 zero-shot vs a 0.46 majority baseline in independent testing. Fine-tune those or don’t trust them.

Why not just use the Jev API? It’s $0.042/M tokens. Latency (236–276ms vs 8–63ms local), privacy, and early-access friction (Jev is waitlisted). At hobby volume the money is a rounding error either way — this is a privacy-and-latency decision, not a cost one.

Does this work in Ollama or LM Studio? Not as of October 25, 2026. LM Studio loads the weights but exposes no decision endpoint; Ollama hasn’t shipped support. You need llama.cpp’s llama-server from the October 2 build onward.

Sources

Last updated October 25, 2026. Prices and specs change; verify current rates before purchasing. GPU prices reflect verified street pricing, not MSRP.

Recommended Gear

RTX 3090 24GB (used) — the cheapest 24GB card; holds a 27B main model and a 4B router resident at once
RTX 5060 Ti 16GB — entry card for a 12B-main + 4B-router stack

Was this article helpful?

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.