TL;DR: llama.cpp v0.7.0 (tagged October 15, 2026) adds --moe-cache-mib, a VRAM cache for MoE experts that live in system RAM. On the pull request author’s own benchmark it takes an RTX 5090 from 30.8 to 67.8 tok/s on a 93.7 GiB model — that’s the 2.2x headline. An RTX 4090 gets 1.62x, AMD cards can get slower, and models that already fit in VRAM gain exactly nothing. The release also moves the default server port from 8080 to 9931, which will silently break your Open WebUI and reverse-proxy configs.

What you’ll be able to do after this guide:

• Decide in 30 seconds whether the MoE cache applies to your model + GPU combination (most 24GB setups running ≤32B models: it doesn’t)
• Size --moe-cache-mib correctly and verify the hit rate instead of copying a random number from Reddit
• Fix every client that breaks when the server moves to port 9931

Honest take: if you run a bigger-than-VRAM MoE on an NVIDIA card — gpt-oss-120b or Qwen3.8-Flash-Next on a 16-24GB GPU — upgrade today, this is the biggest free speedup of the year. Everyone else: upgrade for the model support, but budget ten minutes for the port change.

What actually shipped in v0.7.0

The release landed on October 15, 2026, and the release notes bundle four things that matter for home hardware:

1. MoE expert cache (--moe-cache-mib, PR #29887): host-memory experts get an LRU cache in VRAM, “up to 2.2x faster decode” when experts are offloaded. One day after the initial merge, PR #30112 extended the cache across multiple GPUs.
2. LiliCorr speculative decoding (PR #30506): a candidate-selection layer on top of DFlash drafting. More on why this is not the plug-and-play speedup it sounds like below.
3. Default server port change (PR #30159): llama-server now listens on 9931 instead of 8080. Docker images keep 8080 inside the container, which makes the breakage inconsistent across setups — the worst kind.
4. New model support: K2 Horizon (dense and MoVA variants), Prism Bonsai 2 27B (a ternary model), EmbeddingGemma-2 (text, vision, and audio embeddings), MiniCPM-V 4.7, Nemotron Super 3.5 VL, and a batch of decision models (LiquidAI d1-3B, d1-omni-600M, pplx-decider). If you run a router model next to your main model, our decision-model VRAM guide now applies to mainline llama.cpp without patches.

The MoE cache is the reason to care, so let’s take it apart.

How the MoE cache works (and why it’s not the same as —n-cpu-moe)

The standard way to run a MoE model that doesn’t fit in VRAM has been --n-cpu-moe (or -cmoe for all layers): park the expert weights in system RAM, keep attention and the shared layers on the GPU, and eat the cost of computing expert math on the CPU or shuttling weights over PCIe every single token. We covered that split — and the viral claims around it — in the streaming expert loading piece.

The v0.7.0 cache changes what happens to those host-resident experts. Author am17an (porting the expert cache from the qvac-fabric fork) made the MUL_MAT_ID operations for CPU-side experts run on the GPU, reading expert weights from an LRU cache in VRAM. Only cache misses get uploaded over PCIe. Because MoE routing is heavily skewed — a small set of experts is “hot” for any given prompt — a cache that holds roughly 10% of total expert weights can serve 70-90% of lookups.

Two constraints from the PR worth knowing before you build expectations:

• Small batches only (≤32 tokens). The cache accelerates decode — token-by-token generation. Prompt processing at large batch sizes takes the regular offload path, so your time-to-first-token on a long prompt doesn’t change.
• It needs spare VRAM. The cache competes with model layers and KV cache for the same VRAM pool. If your card is already packed to the last megabyte, there’s nothing left to cache into.

The real numbers: 2.2x is one configuration, not a general rule

Here is the PR author’s own benchmark — Qwen3.8-Flash-Next Q4_0, a 93.7 GiB model with 65.4 GiB of expert weights, on an EPYC 7742 host with PCIe 4.0 x16, running llama-server -fa on -c 32768 -b 2048 -ub 2048 -t 16:

GPU	Configuration	Experts in VRAM + cache	Decode	Speedup	Cache hit rate
RTX 4090 24GB	--fit (v0.6 behavior)	16.5 GB + 0	25.0 tok/s	baseline	—
RTX 4090 24GB	--fit --moe-cache-mib 6544	10.4 GB + 6.4 GB	39.4 tok/s	1.57x	72%
RTX 4090 24GB	-cmoe --moe-cache-mib 11000	0 + 10.7 GB	40.7 tok/s	1.62x	77%
RTX 5090 32GB	--fit (v0.6 behavior)	24.5 GB + 0	30.8 tok/s	baseline	—
RTX 5090 32GB	--fit --moe-cache-mib 6544	17.9 GB + 6.4 GB	54.5 tok/s	1.77x	77%
RTX 5090 32GB	-cmoe --moe-cache-mib 19000	0 + 18.6 GB	67.8 tok/s	2.20x	89%

Source: PR #29887, measured by the PR author. We haven’t reproduced these on our own bench yet.

Read the table honestly and three things fall out:

The headline 2.2x is the RTX 5090 with a 19GB cache and an 89% hit rate. That configuration exists because a 32GB card can afford to dedicate 18.6GB of VRAM purely to expert caching. A 24GB card tops out around 1.6x — still excellent, still free, but it’s not 2.2x.

The counterintuitive result: full offload plus cache beats partial offload plus cache. -cmoe --moe-cache-mib 11000 (no experts statically placed, big cache) outran --fit --moe-cache-mib 6544 (some experts pinned, smaller cache) on both cards. Static placement pins experts whether they’re hot or not; the LRU cache keeps whichever experts the current workload actually routes to. If you adopt the cache, let it do the placing.

Hit rate is the whole game. A commenter on the PR estimated that on a PCIe 4.0 x8 slot, the cache only breaks even with CPU decoding at roughly an 85% hit rate for a large model — below that, upload traffic eats the gain. If your second GPU sits in a chipset x4 slot (the classic mistake from our dual-GPU guides), temper expectations accordingly.

Independent confirmation on budget hardware

One third-party datapoint beyond the PR: developersdigest.tech ran the cache on a free-tier Tesla T4 16GB and measured Qwen3-30B-A3B going from 22.6 to 43.2 tok/s — a 1.9x gain on a card with a fraction of a 5090’s bandwidth (their write-up). That matters because it suggests the mechanism scales down: the win comes from avoiding PCIe round-trips, not from Blackwell-specific hardware.

Who gains nothing (read this before you upgrade for the speedup)

The cache only does work when expert weights live in host RAM. That rules out the most common setup on this site:

• Qwen3.8-27B Q4_K_M on a 24GB card: the GGUF is about 17GB and fits entirely in VRAM. There are no host-resident experts to cache. Your ~41 tok/s stays ~41 tok/s. Any claim that v0.7.0 makes fully-resident models faster is wrong on mechanism.
• Dense models, any size: no experts, no cache. A 70B dense model offloaded to RAM is still bound by the old math.
• AMD cards, today: PR comments report a Vulkan user regressing from 23 to 18.1 tok/s with the cache on, a ROCm user going from 25 to 23 tok/s, and a crash-at-load issue filed against Vulkan. An HIP user saw a small decode gain and no prompt-processing change. If you’re on a 7900 XTX, leave the flag off until the backlog clears.

Who does gain: anyone running gpt-oss-120b, Qwen3.8-Flash-Next, or any big-total/small-active MoE that spills into system RAM on a 16-24GB NVIDIA card. That’s exactly the “180B on disk, 6B active” class of model that made local LLMs good this year — and our Flash-Next guide measured 22-33 tok/s on 24GB cards with plain --n-cpu-moe, which is the baseline this cache improves on.

Upgrading and turning it on

Source builds:

cd llama.cpp
git pull
git checkout v0.7.0
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Binary users: grab the October 15 or later release from the GitHub releases page. Then launch with the cache:

./build/bin/llama-server -m qwen3.8-flash-next-Q4_0.gguf \
  -ngl 99 -cmoe --moe-cache-mib 11000 -fa on -c 32768

Expected output — note the port:

main: server is listening on http://127.0.0.1:9931 - starting the main loop

Sizing rule from the PR author: start at about 10% of total expert size and grow until you run out of spare VRAM. For a model with 65GB of experts that’s ~6,500 MiB; on a 24GB card running -cmoe you can push toward 11,000 MiB since no experts are statically placed. Watch the hit rate the server logs — if you’re stuck under ~75% on a PCIe 4.0 x16 slot (higher threshold on x8), shrink the context or accept that your workload’s routing is too flat to cache well.

The port change will break things the speedup won’t fix

llama-server defaulting to 9931 instead of 8080 is a one-line change (PR #30159) with a long blast radius. After upgrading, check:

• Open WebUI / LibreChat / AnythingLLM: any OpenAI-compatible endpoint configured as http://localhost:8080/v1 now needs :9931, or the UI shows connection-refused.
• Reverse proxies: nginx/Caddy upstream blocks pointing at 127.0.0.1:8080.
• Scripts and cron jobs hitting the completion endpoint directly.
• Docker users: the official images still expose 8080 inside the container, so -p 8080:8080 mappings keep working — which means your laptop setup breaks while your containerized server doesn’t, and vice versa.

The lazy fix is to pin the old port at launch:

llama-server -m model.gguf --port 8080

Verify whichever port you chose before blaming the model:

curl http://localhost:9931/health
# {"status":"ok"}

LiliCorr: real, merged, and not ready to be your speedup plan

The release notes list “LiliCorr speculative decoding: candidate selection with a small draft transformer on top of DFlash” (PR #30506, merged October 15). It implements an NVIDIA technique (arXiv 2608.20530): where DFlash takes the top token at each draft position, LiliCorr runs a small transformer that scores top-k candidates by cosine similarity between learned in/out vectors, aiming for a higher acceptance ratio at some extra compute cost.

Three reasons to keep expectations in check:

1. It requires a dedicated draft checkpoint with the LiliCorr transformer baked in — converted via convert_hf_to_gguf.py with --target-model-dir. It does not work with an arbitrary draft GGUF you already have.
2. The PR publishes no tok/s, acceptance-rate, or speedup numbers. The author reports “similar acceptance length to vLLM” on an RTX 5090 and nothing more.
3. The maintainer flagged the graph as likely unoptimized, with refactoring planned.

If you want measured speculative-decoding gains today, the boring answer is still DFlash2 on Qwen3.8-27B — real flags, measured 1.8-2.7x — or the general llama.cpp speculative setup. Revisit LiliCorr when checkpoints and numbers exist.

Does v0.7.0 change what GPU to buy?

Slightly — in the direction of VRAM still winning, for a new reason. The cache turns spare VRAM into decode speed on oversized MoE models: the 5090’s 2.2x over the 4090’s 1.62x in the table above is almost entirely explained by the 19GB vs 11GB cache budget. A card that fits the whole model still beats any cache configuration, and the bandwidth hierarchy from our 24GB VRAM guide is unchanged.

Practically: if you’re on a 16GB card eyeing the big MoE class, a used RTX 3090 at $1,456-$1,558 (street, verified October 20, 2026) remains the floor for comfortable -cmoe + cache headroom — the RTX 50 SUPER refresh won’t rescue you this year. Not ready to spend? A rented 3090 on Vast.ai from $0.07/hr is enough to test whether your target model’s hit rate makes the cache worthwhile on your workload before any money moves. Check what your own model + context actually needs with the VRAM calculator.

FAQ

Does --moe-cache-mib help if my model fits entirely in VRAM? No. The cache accelerates experts stored in system RAM. A fully VRAM-resident model has none, so the flag allocates memory for zero benefit.

What value should I start with? ~10% of the model’s total expert size, per the PR author. Grow it while VRAM allows; the logged hit rate tells you when you’ve hit diminishing returns.

Does it speed up prompt processing? No — the cache path only engages for batches of ≤32 tokens, i.e. decode. Long-prompt time-to-first-token is unchanged.

Does it work with multiple GPUs? Yes, as of the follow-up PR #30112 the cache is shared across GPUs. The initial merge was single-device only.

Why did my Open WebUI stop responding after upgrading? The server default port moved from 8080 to 9931. Either update your endpoint URL or launch with --port 8080.

Should AMD users enable it? Not yet. Reported results on Vulkan and ROCm range from slightly slower to crash-at-load. Watch the issue tracker before enabling.

Is this the same as Ollama’s MoE handling? No — Ollama hasn’t shipped this cache. If you front llama.cpp with coding tools, a BYOK setup pointed at llama-server (see aicoderscope.com for the Cursor/Cline side, and aifoss.dev’s Ollama review for where Ollama stands) picks up the speedup today.

Sources

• llama.cpp v0.7.0 release notes — GitHub, ggml-org
• PR #29887: llama: add a GPU cache for MoE experts kept in host memory — GitHub, am17an
• PR #30112: share the MoE expert cache across multiple GPUs — GitHub
• PR #30159: server: change default port to 9931 — GitHub
• PR #30506: LiliCorr speculative decoding — GitHub
• LiliCorr paper — arXiv 2608.20530, NVIDIA
• llama.cpp MoE Offload: —n-cpu-moe vs the New Expert Cache — Developer’s Digest
• llama.cpp 0.7.0: MoE cache hits 2.2x, server port has changed — byteiota

Last updated October 27, 2026. Prices and specs change; verify current rates before purchasing.

Get the numbers before you buy

New GPU and mini-PC benchmarks, VRAM thresholds, and price checks — sent when there's something worth acting on, not on a schedule. No spam, unsubscribe anytime.

Email address Subscribe