TL;DR: A 26-billion-parameter model running at a usable 5 tok/s on a 2013-era Xeon with zero GPU sounds impossible — until you notice Gemma 4 26B is a Mixture-of-Experts model that only activates ~3.8B parameters per token. That is the whole trick. CPU inference is viable here because of the model, not the hardware.

What you’ll be able to do after this guide:

• Understand which CPU-only builds can actually run a 26B-class model, and why memory bandwidth — not core count — is the ceiling.
• Build and run ik_llama.cpp on an old server, including the AVX1 fallback that trips up pre-2015 Xeons.
• Decide honestly whether a $200 used server or a $1,254 used RTX 3090 is the right buy for your workload.

Honest take: Do this if you already own the hardware or want an always-on tinker box for under $250. If you’re buying from scratch and speed matters, a used RTX 3090 runs the same model ~15× faster for real money — CPU inference wins on cost and silence, never on tokens per second.

The claim that made the rounds

In June 2026, Neomind Labs published a write-up titled “Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU.” It hit the Hacker News front page, and the instinct in the comments was disbelief: a 26B model, on a decade-old server, with no accelerator, generating text at a readable pace?

The setup was a repurposed HP StoreVirtual storage appliance — dual Intel Xeon E5-2690 v2 (Ivy Bridge, 2013, 10 cores each / 20 total), 128 GB of slow DDR3, and no GPU. Running Gemma 4 26B at Q8_0, the author measured roughly 5.2 tok/s for decode and ~16 tok/s for prompt evaluation.

That is genuinely usable for chat and structured tasks — human reading speed is about 7–10 tok/s, so 5 tok/s feels slow but never frustrating for short replies. The interesting part is why it works, because the answer changes what you should buy.

Why this works: it’s the MoE, not the magic

The headline hides the load-bearing detail. Gemma 4 26B is not a dense 26B model. Its full name is Gemma 4 26B-A4B — a Sparse Mixture-of-Experts model with 26.1B total parameters but only ~3.8B active per token (128 experts, 2 selected per token). The “A4B” literally means “Active 4 Billion.”

That distinction is everything for CPU inference. On a CPU, token generation is almost entirely memory-bandwidth-bound: for every token, the machine streams the active weights from RAM through the cores. A dense 26B model at Q8_0 would need to read ~26 GB per token. Gemma 4 26B-A4B only reads the ~4 GB worth of active experts per token.

Do the arithmetic against the hardware. The Xeon E5-2690 v2 has a maximum memory bandwidth of 59.7 GB/s across its four DDR3 channels (Intel spec). Real-world sustained bandwidth is lower — call it 40–50 GB/s — but even at ~4 GB read per token that’s a theoretical ceiling around 10–12 tok/s, landing at ~5 tok/s once you account for attention, KV cache, and quantization overhead. A dense 26B on the same box would crawl at well under 1 tok/s.

This is the takeaway most CPU-inference posts bury: the entire strategy depends on running a MoE model with a small active-parameter count. Try the same trick with a dense 27B or a 70B and the bandwidth wall stops you cold. If you want to run big models on CPU, pick the architecture first.

What hardware actually qualifies

You do not need a rare unicorn server. You need three things, in priority order:

1. Enough RAM to hold the whole model. Even though only ~4B params are active per token, all 26.1B must live in memory. Gemma 4 26B footprints: ~14–16 GB at Q4_K_M, ~28 GB at Q8_0, ~52 GB at BF16. So a 32 GB machine runs Q8_0 comfortably; 16 GB runs Q4 with room for context.
2. The most memory bandwidth you can get. This is the real speed knob. Quad-channel DDR3-1866 (~60 GB/s) is the floor that made the Xeon work. A modern dual-channel DDR5-6000 desktop hits ~96 GB/s — roughly 1.6× the old Xeon, so expect proportionally faster decode. Server platforms with 8- or 12-channel DDR5 (EPYC, Xeon Scalable) are where CPU inference gets genuinely fast.
3. Cores, last. Prompt processing (the 16 tok/s number) scales with cores and benefits from AVX. Decode barely cares past ~8 threads because it’s bandwidth-bound, not compute-bound.

For a home lab, the sweet spot is a used dual-socket server. A pair of Xeon E5-2690 v2 CPUs sells used for well under $100, and complete LGA2011 servers with 128 GB DDR3 routinely go for $150–$300 on eBay — cheaper than a single RTX 3060 12GB. If you’re speccing new for bandwidth, a desktop with fast DDR5 and a modern Corsair Vengeance 64GB DDR5 kit will comfortably outrun the old Xeon on the same model.

Setting up ik_llama.cpp

Vanilla llama.cpp runs Gemma 4 fine, but the 5 tok/s number came from ik_llama.cpp — ikawrakow’s MIT-licensed fork that adds fused MoE operations, SOTA quantization types, and faster CPU kernels. For MoE models on CPU, the fork’s fused expert path is exactly the optimization that matters.

Basic build on a modern CPU:

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j $(nproc)

Then grab a GGUF and run it. Pin the thread count to your physical cores and give it a sane context:

./build/bin/llama-cli \
  -m gemma-4-26b-a4b-Q8_0.gguf \
  -t 20 \
  -c 8192 \
  -p "Explain memory bandwidth in one paragraph."

On a fast DDR5 desktop this is all you need. The complication is old hardware.

The AVX1 gotcha (pre-2015 Xeons)

Here is the part that generated the actual blog post. ik_llama.cpp assumes AVX2 as its floor — its high-performance quantized kernels (the IQK path) are written for AVX2 and better. The Xeon E5-2690 v2 has AVX1 but not AVX2. Build normally and it either fails to compile the fast paths or produces garbled output.

The fix (documented in ik_llama.cpp PR #2138) is to disable the AVX2-only kernels so the code falls back to plain scalar/SSE math:

cmake -B build -DGGML_NATIVE=ON -DGGML_USE_IQK_MULMAT=OFF

Two more rules for AVX1 boxes:

• Drop --run-time-repack. It reorders weights into an AVX2-only memory layout and will re-garble output on AVX1.
• Prefer Q8_0 over the fancy IQ quants. The trellis/IQK quant types lean hardest on the AVX2 kernels you just disabled; Q8_0 is simple enough to stay fast on the scalar path, which is why the original run used it.

If your CPU is 2015 or newer (Haswell/Broadwell Xeon E5 v3/v4, any Ryzen, any Core i-series from the last decade), you have AVX2 and none of this applies — build normally and use the IQ quants for a smaller footprint.

When CPU inference actually makes sense

Be honest with yourself about the tradeoff. Here’s how the same Gemma 4 26B-A4B run stacks up across three realistic home-lab options:

	Old Xeon (CPU only)	Radeon 780M iGPU	Used RTX 3090
Best for	Always-on, silent, near-free	Mini-PC / laptop dabbling	Anyone who wants speed
Gemma 4 26B decode	~5 tok/s (Q8_0)	~23–25 tok/s (Q4, Vulkan)	~76 tok/s (Q4)
Cost	~$150–$300 used server	already in the machine	~$1,254 used (Jul 2026)
The catch	bandwidth wall; MoE-only	16 GB shared-RAM ceiling	350W, needs a real PSU

The RTX 3090 number (~76 tok/s) is roughly 15× faster than the Xeon, so if you generate long outputs or run an agent loop, the GPU pays for itself in wall-clock sanity. But CPU inference wins three specific cases:

• Always-on background jobs. Document extraction, webhook summarization, nightly batch classification — none of these care about latency, and an idle old server sips power.
• You already own the box. A retired server or a workstation with 64 GB+ of RAM is free inference capacity. Running Gemma 4 26B on it costs you nothing but electricity.
• Model size exceeds any GPU you can afford. A 128 GB RAM server can hold models that would need multiple GPUs. Slow-but-possible beats fast-but-impossible.

If none of those fit — if you want interactive speed and you’re buying hardware anyway — rent before you commit. A few hours on a cloud GPU via RunPod will tell you whether Gemma 4 26B is even the right model for your task before you spend $250 on a server or $1,254 on a card. For the full CPU-vs-GPU build breakdown, see our best CPU for AI workstations guide, and for picking a model that fits your memory budget, the best local AI models by VRAM tier.

FAQ

Is Gemma 4 26B really a 26-billion-parameter model? Yes — 26.1B total parameters, but it’s a Mixture-of-Experts model that activates only ~3.8B per token (128 experts, 2 active). That’s why it runs on modest hardware: memory bandwidth per token is set by the active count, not the total.

Will a dense 26B or a 70B model also run at 5 tok/s on CPU? No. A dense 26B streams all ~26 GB per token instead of ~4 GB, so on the same Xeon you’d see well under 1 tok/s. The 5 tok/s result is specific to small-active-parameter MoE models. Match the architecture to the strategy.

How much RAM do I need? Enough to hold the whole model plus context: ~14–16 GB for Q4_K_M, ~28 GB for Q8_0, ~52 GB for BF16. 32 GB is a comfortable floor for Q8_0; the original build used 128 GB, which is overkill for this model but leaves room for long context and bigger MoE models.

My CPU is older than 2015 — will ik_llama.cpp work? If it lacks AVX2 (like the Xeon E5-2690 v2), build with -DGGML_USE_IQK_MULMAT=OFF, skip --run-time-repack, and stick to Q8_0. Anything Haswell-era (2014–2015) or newer has AVX2 and needs no workaround.

Should I buy a GPU instead? If interactive speed matters and you’re buying anyway, yes — a used RTX 3090 runs this model ~15× faster. CPU inference is for people who already own the hardware, want an always-on silent box, or need to run models too large for their GPU budget. See our Gemma 4 GPU guide for the accelerated path.

Does more RAM speed make a difference? Directly, yes — decode speed tracks memory bandwidth almost linearly. Quad-channel DDR3-1866 gave ~60 GB/s. Faster DDR4/DDR5 or more channels (8-/12-channel server platforms) is the single biggest lever on CPU token rate. See our note on how the 2026 DDR5 price surge affects build costs.

Sources

1. Neomind Labs — “Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU” (Jun 8, 2026): https://www.neomindlabs.com/2026/06/08/running-gemma-4-26b-at-5-tokens-sec-on-a-13-year-old-xeon-with-no-gpu/
2. Hacker News discussion (item 48922434): https://news.ycombinator.com/item?id=48922434
3. ik_llama.cpp — ikawrakow’s llama.cpp fork (MIT, better CPU + MoE performance): https://github.com/ikawrakow/ik_llama.cpp
4. Google — Gemma 4 26B-A4B model card (26.1B total / ~3.8B active MoE, 128K context): https://huggingface.co/google/gemma-4-26B-A4B
5. Intel — Xeon E5-2690 v2 product specifications (59.7 GB/s max memory bandwidth, quad-channel DDR3): https://www.intel.com/content/www/us/en/products/sku/75279/intel-xeon-processor-e52690-v2-25m-cache-3-00-ghz/specifications.html
6. ggml-org/llama.cpp Discussion #24222 — Gemma 4 26B on Radeon 780M iGPU via Vulkan (~23–25 tok/s): https://github.com/ggml-org/llama.cpp/discussions/24222
7. Google AI for Developers — Gemma 4 model overview: https://ai.google.dev/gemma/docs/core

---

Prices and benchmarks verified as of July 2026 and will drift — used hardware pricing especially moves week to week. Building a local coding setup on top of a model like this? See aicoderscope.com for connecting local models to Cline and Cursor, and aifoss.dev for the open-source ik_llama.cpp deep dive.

Recommended Gear

• RTX 3090 — the ~15×-faster GPU alternative; best bandwidth-per-dollar under 24GB.
• RTX 3060 12GB — cheapest CUDA card that fits Gemma 4 26B at Q4.
• Corsair Vengeance 64GB DDR5 — fast dual-channel RAM if you’re building a modern CPU-inference box.