Tencent Hy3 for Local AI in 2026: The First 295B Frontier MoE That Fits a $2,000 Home-Lab Box

By RunAIHome Team · 12 min read
tencenthy3local-llmmoellama-cppquantizationhardware-guideopen-weights

TL;DR: Tencent released Hy3 — a 295B/21B-active MoE, Apache 2.0, 256K context — on July 6, 2026, then shipped official 1-bit and 4-bit GGUFs on July 14 alongside a merged llama.cpp architecture. The 1-bit build is 91.8GB: it runs on a $1,999 128GB Strix Halo mini PC at a measured 24.3 tok/s with MTP speculative decoding enabled.

Hy3 IQ1_M local (128GB box)Hy3 via APIKimi K2.7 local (2-bit)
Best forPrivate frontier-class agent on one quiet boxCasual and metered useMax open-weight coding quality
Price / CostGMKtec EVO-X2 128GB ~$1,999–$2,199$0.14 in / $0.58 out per M (OpenRouter)~325GB → $4,000+ in RAM/GPUs
The catch1-bit quant quality loss is real and mostly unmeasuredPrompts leave your machine3× the memory, single-digit tok/s

Honest take: Hy3 is the first frontier-class open model whose official quant fits hardware a normal person can buy — one 128GB unified-memory box, no GPU farm. If you own a Strix Halo machine or a 128GB Mac, download it this weekend. If you don’t, the API at $0.58/M output is so cheap that buying hardware just for Hy3 only makes sense for privacy or always-on agent workloads.

Every big open-weight release this summer has ended the same way for home labs: GLM-5.2 needs ~239GB at 2-bit, Kimi K2.7 needs ~325GB, Inkling needs ~290GB at 1-bit. The verdict was always “use the API.” Hy3 breaks the pattern — not because the model is small, but because Tencent did the quantization work itself and targeted a memory budget that real machines actually have.

What Tencent shipped, and when

Two separate drops matter here:

July 6, 2026 — Hy3 proper: 295B total parameters, 21B active per token (192 experts, top-8 routing), 80 transformer layers plus one 3.8B NextN/MTP layer, 256K context, trained for reasoning and agentic coding. License is Apache 2.0 with no geographic or field-of-use restrictions — a meaningful upgrade from the April Hy3-preview, which carried regional limits. Weights went up on Hugging Face, and OpenRouter ran a free tencent/hy3:free route through July 21.

July 14, 2026 — the part that changes your hardware math. Tencent released official 1-bit and 4-bit GGUF builds, and the same day the hy_v3 architecture merged into mainline llama.cpp (PR #25395). Any master build from July 14 onward (≥ b10005) loads the files directly — including the bundled MTP layer for speculative decoding. Tencent’s own framing: a flagship-scale model “served on a single GPU.”

One important decode on that “single GPU” claim: they mean a 128GB-class device — a Strix Halo mini PC, a DGX Spark, a 128GB Mac — not a single RTX card. No consumer GPU holds 91.8GB. But 128GB unified-memory boxes start at $1,999 now, and that’s the cheapest ticket to a 295B model anyone has offered yet.

The GGUF ladder: what fits where

Verified sizes from the official release and community builds (MTP layer adds ~2GB and is worth keeping — more below):

QuantSize (with MTP)Fits onNotes
IQ1_M (official 1-bit)91.8GB128GB Strix Halo / DGX Spark / 128GB MacThe headline build; quality floor mostly unmeasured
IQ2_M (community)~93GBSame 128GB classBetter quality per GB than IQ1_M in most models
Q2_K / Q2_K_XL~101–102GB128GB Linux boxes (tight), 24GB GPU + 128GB RAM offloadGTT allocation needed on Strix Halo
Q3_K_M~136GB192GB+ setups, 256GB RAM CPU-offload builds
Q4_K_M (official 4-bit)~169GB2× RTX PRO 6000 (192GB), 256GB RAM builds, Mac Studio clusterTencent calls this “near full performance”

For context on how favorable this ladder is: at 2-bit, Hy3 is ~102GB where GLM-5.2 is ~239GB and Kimi K2.7 is ~325GB. Tencent’s 295B lands at roughly the memory footprint the 100B-120B dense class occupied a year ago.

The KV cache is the other budget line. At 8K–16K context on a 128GB box running IQ1_M you have ~20–30GB of headroom, which is comfortable. Don’t plan on the full 256K context locally — that’s an API-tier luxury, same story as every big-context model we’ve covered.

Measured speed: 24.3 tok/s on a $2,000 mini PC

The numbers that make this article worth writing come from community benchmarks on AMD Strix Halo (Ryzen AI Max+ 395, 128GB LPDDR5X at 256 GB/s) — the chip inside the GMKtec EVO-X2 we reviewed last month:

  • Baseline decode: ~17 tok/s on Hy3 quantized GGUF via llama.cpp
  • With MTP enabled: 24.3 tok/s on code (+40%), ~20 tok/s on prose
  • Draft acceptance: 88–97% on code with the right flags, ~91% typical at temp 0.9

That’s a 295B-class model decoding above 20 tok/s — roughly 2–3× human reading speed — on a machine that idles at 8–14W. For comparison, the same Strix Halo box runs dense Llama 3.3 70B at ~5 tok/s, and a $3,999 DGX Spark decodes 70B at ~2.7 tok/s. The MoE math is the whole story: 21B active parameters means each token reads ~12–13GB of weights at 1–2 bit instead of all 295B, so a 256 GB/s memory bus is enough. We walked through why sparse MoE plus speculative decoding is the 2026 efficiency story in Why Local LLMs Got Good; Hy3 is that thesis shipping as a product.

DGX Spark (273 GB/s) should land within ~5–10% of the Strix Halo numbers, and a 128GB M4-class Mac (546 GB/s on Max chips) should roughly double them — both are bandwidth-scaled estimates, not measurements; no published runs yet as of July 25.

Setting up MTP — and the flag that makes or breaks it

Hy3 ships its speculative-decoding drafter inside the GGUF: the NextN/MTP layer (blk.80) is bundled, so there’s no separate draft model to download, unlike the classic two-model setup in our speculative decoding guide. You enable it with the 2026 flag names:

# llama.cpp master ≥ b10005 (July 14, 2026)
./llama-server -m Hy3-Instruct-IQ1_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-p-min 0.75 \
  -ngl 99 -c 8192 -fa on

Expected log lines when it’s working:

load: MTP/NextN layer found: blk.80
spec: draft-mtp enabled, p-min = 0.75
...
eval: 24.31 tokens per second, draft acceptance 91.2%

The real problem you’ll hit: run it with default settings and Hy3’s draft acceptance collapses to ~39% — the speculation overhead eats the gain and you decode barely faster than baseline, sometimes slower. The fix is --spec-draft-p-min 0.75, which stops the MTP head from proposing low-confidence drafts; acceptance jumps to 88–97% and the +40% throughput shows up. This tripped up nearly everyone in the first week, because no other MTP model (Qwen3.6, Gemma 4) needs the p-min raised this aggressively. If your tokens/sec didn’t move after enabling MTP, this flag is why.

Two more first-week traps, both familiar from other 2026 releases:

  1. unknown model architecture: 'hy_v3' — your llama.cpp (or Ollama, or LM Studio runtime) predates July 14. Rebuild from master or update; the diagnosis steps here apply verbatim. Ollama had no managed Hy3 tag as of July 25, so llama.cpp is the maintained path this week.
  2. Buying the no-MTP GGUF to save 2GB. The MTP-less files are ~2GB smaller (89.4 vs 91.8GB at 1-bit) and forfeit the 40% speedup. Keep the MTP layer unless you literally cannot fit it.

Which hardware actually makes sense

Own a 128GB Strix Halo box (EVO-X2, Ryzen AI Halo kit, Framework Desktop): download IQ1_M today. This is the best model your machine has ever run, and the capacity-over-speed bet we made in the EVO-X2 review just paid out. The head-to-head with DGX Spark tilts further toward whichever box is cheaper — both run Hy3 at similar speed.

Own a 24GB GPU + 128GB system RAM: the Q2_K path with expert offload works — -ot ".ffn_.*_exps.=CPU" keeps attention and shared layers on the GPU and streams experts from RAM. Expect single digits to low teens tok/s depending on DDR5 speed (estimate — no published Hy3 runs on this config yet; the pattern matches Qwen3-Coder 480B offload behavior). A used RTX 3090 (~$1,050 eBay low, ~$1,250 marketplace average, July 2026) plus a RAM upgrade is the scrappy version of this build — but note 128GB of DDR5 costs real money in the current memory market.

Own nothing yet and want Hy3 local: the EVO-X2 128GB at $1,999–$2,199 is the cheapest single-box path. That’s a full working setup for less than two used 3090s, and it pulls 147–160W under load instead of 700W.

Want Q4 quality: you’re in 192GB+ territory — 2× RTX PRO 6000 Blackwell (~$24,000), a 256GB DDR5 server build, or a Mac Studio cluster. At that budget, test on RunPod first — two H100s run Q4 for ~$5.78/hr, which buys a lot of evaluation before you commit five figures.

Quality: where Hy3 sits in the July 2026 field

Tencent’s reported benchmarks (vendor numbers — independent verification is still thin, the standard caveat):

BenchmarkHy3 (full precision)Best open-weight comparison
SWE-Bench Verified78.0Kimi K2.6: 80.2 / Inkling: 77.6
SWE-Bench Pro57.9GLM-5.2: 62.1
SWE-Bench Multilingual75.8—

Read that honestly: Hy3 is in the frontier open-weight pack, not on top of it. GLM-5.2 beats it on SWE-Bench Pro; K2.6 edges it on Verified. What no competitor matches is the deployment story — nothing else in that table runs on a $2,000 box. And the numbers above are for full precision served over API; the IQ1_M build you’d run locally gives up quality that nobody has rigorously measured yet. Tencent claims 4-bit is “near full performance” but published no 1-bit evals — treat 1-bit Hy3 as “frontier-adjacent,” verify on your own workload, and check our July leaderboard for how fast this table goes stale. Kimi K3’s weights land July 27 and may reshuffle it within the week.

If your GPU is a normal 24GB card, the right answer hasn’t changed: Qwen3.6-35B-A3B at ~22GB Q4 remains the single-card champion, and running a 295B via offload is a batch-work tool, not a daily driver.

API vs local: the math

OpenRouter’s post-promo pricing is $0.14/M input, $0.58/M output (the free route expired July 21). At 10M output tokens a month that’s $5.80 — a $2,099 EVO-X2 takes about 30 years to pay back on token costs alone. Pure economics say use the API, same conclusion as our rent-vs-buy analysis.

The local case is everything that isn’t cost-per-token: prompts that never leave your network, an always-on agent that doesn’t meter, offline capability, and Apache 2.0 weights you can fine-tune and ship commercially — no ambiguity, no revenue caps, which our sister site aifoss.dev covers as its own story. If you’re wiring Hy3 into a coding agent as a Claude/Cursor alternative backend, aicoderscope.com covers that setup side.

FAQ

Can an RTX 4090 or 5090 run Hy3? Not alone. The smallest build is 89.4GB; a 32GB RTX 5090 holds about a third of it. A 24GB card contributes usefully only in the CPU-offload configuration with 128GB+ system RAM.

Does Ollama support Hy3? Not as a managed model as of July 25, 2026. The hy_v3 architecture is in mainline llama.cpp (≥ b10005), so llama.cpp directly — or any frontend that ships an updated runtime — is the path this week.

Is the 1-bit quant actually usable? Early community reports on code and agent tasks are positive, and MoE models degrade more gracefully at low bits than dense ones. But there are no rigorous 1-bit evals published. Test against the API (same model, full precision, $0.58/M out) to measure what the quant costs you on your own tasks.

Should I wait for Kimi K3 on July 27 instead? K3 is 2.8T parameters — roughly 10× Hy3’s footprint, ~1.4TB at its native 4-bit. It will not fit a 128GB box under any quant. Different weight class, literally; Hy3 is the one built for hardware you can own.

What context length is realistic locally? 8K–16K comfortably on a 128GB box at 1-bit, more if you quantize the KV cache. The advertised 256K needs server-class memory.

Sources

Last updated July 25, 2026. Prices and specs change; verify current rates before purchasing.

Was this article helpful?