IFM K2 Horizon for Local AI in 2026: Six Fully Open Models — Which GPU Runs the 32B and 36B-A4B?

By RunAIHome Team · 12 min read
k2-horizonifmmbzuaiopen-weightslocal-llmmoe24gb-vramapache-2-0

TL;DR: On September 3, MBZUAI’s Institute of Foundation Models released K2 Horizon — six Apache 2.0 models from 0.9B to 375B-A23B, with training data, code, and methodology included, not just weights. Two of them target 24GB consumer cards. But mainline llama.cpp can’t load any of them yet, and the local testing that exists says these are early checkpoints with real rough edges.

K2-Horizon-MoVA-36B-A4BK2-Horizon-32B (Stage 1)K2-Horizon-375B-A23B
Best for24GB cards; fast MoE decode24GB cards; dense quality, 512K contextNobody’s home lab — API only
Memory needed~18–20GB at 4-bit (est.); 74.9GB BF16~19–20GB at 4-bit (est.); 69.6GB BF16Datacenter scale; Cerebras/Nebius/Compass APIs
The catchNeeds IFM’s llama.cpp fork; no upstream merge yetIt’s a Stage 1 checkpoint — the final isn’t outFork intentionally doesn’t support it

Honest take: Don’t buy hardware for K2 Horizon, and don’t make it your daily driver this week — early local tests found real bugs, and the toolchain is a fork. Watch it anyway: this is the only 24GB-class model family where you can inspect every training token, and if the final checkpoints close the quality gap, “fully open” stops being a research curiosity and becomes a real Qwen alternative.

First, the name collision, because 2026 has been cruel here: K2 Horizon has nothing to do with Moonshot’s Kimi K2, the 1T-parameter MoE we covered in June. This K2 comes from the Institute of Foundation Models (IFM), the research arm of MBZUAI in Abu Dhabi — the same lineage as the LLM360 K2-65B project that pioneered “360-degree open” releases. On September 3 they shipped six models at once under Apache 2.0, and unlike nearly every “open” release this year, the package includes the training data and training code, not just a weights dump.

For home labs, two of the six matter: the 32B dense and the 36B-A4B sparse MoE, both of which quantize into a single 24GB card. Here’s the real hardware math, what’s verified versus estimated, and the toolchain catch that decides whether you can run them at all this week.

The six models, sorted by where they actually run

IFM sized each model for a different deployment target, from wearables to racks. Each was pretrained on roughly 20 trillion tokens — about 10 trillion of them synthetic — then post-trained on more than 100 million generated tasks.

ModelTypeContextIFM’s targetHome-lab reality
K2-Horizon-0.9BDense128KWatches, glasses~2GB VRAM; runs on anything, even CPU
K2-Horizon-3.7BDense—PhonesAny 6GB+ card
K2-Horizon-7BDense512KPhones, on-device~17GB BF16; ~4.5GB at Q4 (est.) → 8GB cards
K2-Horizon-32B (Stage 1)Dense512KLocal hosting, on-prem24GB card at 4-bit
K2-Horizon-MoVA-36B-A4BMoE + MoVA—Local hosting24GB card at 4-bit, MoE speed
K2-Horizon-375B-A23BSparse MoE512KDatacenterAPI only (Cerebras, Nebius, Compass)

The context windows and VRAM figures come from IFM’s launch post: 128K context on the 0.9B, 512K on the 7B, 32B, and 375B, with the 0.9B needing about 2GB, the 7B about 17GB in BF16, and the 32B about 92GB with KV cache at a 65K context when served unquantized through vLLM. That last number is the server-deployment figure — the consumer story runs through GGUF quants, and that’s where it gets interesting.

The 24GB math: both mid-size models fit, barely

The official IFM/K2-Horizon-32B-GGUF repo lists the BF16 artifact at about 69.6GB; the MoVA-36B-A4B BF16 GGUF is about 74.9GB. At 4-bit, the raw weight floor for the 32B works out to roughly 18GB, and community guidance for the 36B-A4B puts 4-bit weights at 18–20GB — both inside a 24GB card for the weights alone, with 4–6GB left for KV cache and runtime overhead. Treat the Q4_K_M figures as estimates until the quant files settle; the pattern matches every 32B-class GGUF we’ve measured, where 4-bit lands at 19–20GB and leaves room for 8–16K of context.

What that means in practice, tier by tier:

  • 8–12GB cards (RTX 3060, 4060, 5060): the 7B at Q4 (~4.5GB, estimated) is your ceiling. The 32B and 36B don’t fit at any usable quant.
  • 16GB cards (RTX 5060 Ti, 4060 Ti 16GB, 5080): still no. An 18–20GB 4-bit file doesn’t fit in 16GB — same wall as Qwen3.6-27B at this tier.
  • 24GB cards (used RTX 3090, RTX 4090): both mid-size models fit at 4-bit with modest context. This is the tier the release actually addresses.
  • 32GB+ (RTX 5090, R9700): room for 5-bit quants or long context, nothing new unlocks.

The 36B-A4B is the more interesting file. It’s a Mixture-of-Experts model that stores 36B parameters but activates only about 4B per token, wrapped in what IFM calls Mixture-of-Values (MoVA) attention — value selection happens through specialized routing rather than standard attention, per the llama.cpp support discussion. On paper that’s the same recipe that makes Qwen3.6-35B-A3B our favorite 24GB daily driver: total parameters set what you store, active parameters set how fast it decodes.

Speed, clearly labeled as estimates: nobody has published K2 Horizon tokens-per-second on consumer RTX cards yet. By bandwidth math, a ~4B-active MoE at 4-bit on a used RTX 3090 (936 GB/s) should land in the same band as the measured MoEs of its class — we’ve measured Qwen3.6-35B-A3B at 107 tok/s and Nemotron-Cascade 2 30B-A3B at 187 tok/s on that card — but MoVA’s routing overhead is an unknown, so treat anything from 80 to 150 tok/s as plausible and nothing as confirmed. The dense 32B at 4-bit is bandwidth-bound like any dense 32B: expect the 35–45 tok/s a 3090 delivers on that class, and roughly 60 on a 4090. The one measured local number in the wild so far is the smallest model: the 0.9B, quantized to a 632MB file, ran at 14 tok/s on CPU — no GPU involved.

The catch: your llama.cpp doesn’t know this architecture

This is the paragraph that saves you a 20GB download. K2 Horizon uses a new k2_horizon architecture tag, and as of September 8, mainline llama.cpp does not support it. On a stock build, the load fails on the unrecognized architecture:

$ llama-server -m K2-Horizon-32B-Q4_K_M.gguf
llama_model_load: error loading model: unknown model architecture: 'k2_horizon'
llama_load_model_from_file: failed to load model

That error means you’re on the wrong build, not that your download is corrupt. The working path today is IFM’s own llama.cpp fork:

git clone --branch model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j
./build/bin/llama-server -m K2-Horizon-32B-Q4_K_M.gguf -ngl 99 -c 16384

The pre-release support discussion spells out the fork’s coverage: the dense 0.9B/3.7B/7B/32B and the MoVA 36B work; the 375B sparse MoE is intentionally unsupported. Two extra wrinkles from the same thread: the 3.7B and 7B were exported against a different transformers version than llama.cpp pins, which is causing conversion friction, and the technical report explaining MoVA in full is still forthcoming.

The Ollama situation needs the same skepticism we applied to GLM-5.3-Flash’s launch. IFM’s release materials name vLLM, SGLang, and Ollama as day-zero serving paths — but Ollama inherits model support from its llama.cpp-lineage runtime, and with the upstream PR unmerged, don’t expect ollama pull to produce a working local K2 Horizon until the architecture lands. If a library tag appears before that merge, check whether it’s actually running on your GPU before you trust it. The genuinely working day-zero paths are the fork above and server-grade stacks: the published vLLM recipe for the 32B is a 2-GPU tensor-parallel BF16 deployment validated on H200-class hardware — real, but not a home-lab config.

Benchmarks: the 36B-A4B punches up, the checkpoints are early

The headline results are genuinely competitive for the weight class. On MindStudio’s compilation of the release benchmarks, the 36B-A4B scores 80.8 on GPQA Diamond — roughly level with Qwen3.6-35B-A3B — and 25.2 on Humanity’s Last Exam, ahead of both Qwen3.6-35B-A3B (17.5) and Gemma 4 31B (23.6). On agentic work it posts 58.6 on Terminal-Bench 2.1 and 26.8 on tau3-Banking.

BenchmarkK2-MoVA-36B-A4BQwen3.6-35B-A3BGemma 4 31B
GPQA Diamond80.8~81 (level)84.3
Humanity’s Last Exam25.217.523.6
Terminal-Bench 2.158.6——

Two large grains of salt. First, these are release-day numbers from the vendor’s evaluation, not independent runs — the same caveat we attached to Ornith and Kimi K2.7. Second, the only hands-on local testing published so far is MindStudio’s, and they titled it “results are rough”: coding and multilingual tasks on the 0.9B, 7B, and 32B surfaced real bugs, consistent with what the model cards admit — the 32B is explicitly a Stage 1 checkpoint with the final still to come.

That “still to come” applies to the openness claim too, and it’s worth being precise because the openness is the whole pitch. As CellCog’s close read of the model cards documents: the 3.7B and 7B ship complete — weights, data, recipe, code. The 36B-A4B and 375B-A23B cards say intermediate checkpoints, data, and training code “will be released.” So today, September 8, the honest description is: the largest fully open release in progress, with the small models fully delivered and the interesting-for-home-labs models at weights-plus-promises. That’s still more than Meta, Google, or Alibaba give you — Apache 2.0 with a public data pipeline versus weights under a custom license — but hold the confetti until the 36B’s training data actually lands.

Should you buy anything for this? No — and here’s the cheap way to try it

Nothing in K2 Horizon justifies new hardware today. If you already own a 24GB card, the 36B-A4B is a free weekend experiment via the fork. If you don’t, the calculus is the same as it was last week: a used RTX 3090 is the cheapest ticket into the 24GB tier that both K2 mid-size models (and everything else we cover) target — verified eBay sold prices averaged about $972 through late August, while active asks now run $1,287–$1,411, averaging $1,343 and up 11.3% in 90 days — the sold-versus-ask gap means patience at the checkout still pays. A used RTX 4090 buys roughly double the prefill speed for agentic loops at about double the price. Run your own model-fit numbers in our VRAM calculator before spending anything.

If you’d rather test before owning — or want the 375B — rent first. A RunPod instance with a 24GB or 48GB card lets you benchmark the 36B-A4B against your current daily driver for a couple of dollars, and the 375B-A23B is reachable through Cerebras and Nebius APIs without touching your rig. For the coding-assistant angle — whether the 7B makes sense as a local Continue.dev/Cline backend — that’s aicoderscope.com territory, and the license-and-reproducibility deep dive on what “training data included” means for auditing belongs to aifoss.dev.

The strategic reason to care isn’t this checkpoint — it’s the precedent. Every fine-tune you build on Qwen or Gemma sits on training data you can’t inspect and a license that isn’t Apache 2.0 all the way down. A 24GB-class model where the data, recipe, and code are public is the first credible answer to that problem, and our fine-tuning cost math applies unchanged to a 36B-A4B QLoRA run. If the final checkpoints close the quality gap with the models that currently own the 24GB tier, K2 Horizon becomes the default recommendation for anyone who needs provenance — regulated industries, research, anyone shipping a product on top of open weights.

FAQ

Is K2 Horizon related to Kimi K2? No. Kimi K2 is Moonshot AI’s 1T-parameter MoE. K2 Horizon comes from MBZUAI’s Institute of Foundation Models in Abu Dhabi, continuing the LLM360 K2-65B “fully open” lineage. Same letter, different continent, different model.

Can I run any K2 Horizon model on an 8GB card? The 7B at 4-bit (~4.5GB, estimated) should fit an 8GB card with short context, and the 0.9B and 3.7B run almost anywhere — the 0.9B’s quantized file is 632MB and manages 14 tok/s on CPU alone. The 32B and 36B-A4B need 24GB.

Does ollama pull work for K2 Horizon? Not usefully yet. Ollama support depends on the k2_horizon architecture landing in the llama.cpp lineage, and that PR hasn’t merged. Until then, the working local path is IFM’s llama.cpp fork.

Why prefer the 36B-A4B over the 32B? Speed and checkpoint maturity. The MoE activates ~4B parameters per token, so it decodes several times faster than the dense 32B at the same 4-bit footprint — and the 32B is explicitly a Stage 1 checkpoint. Pick the 32B only if you need its 512K context and dense-model consistency.

Sources

Last updated September 8, 2026. Prices and specs change; verify current rates before purchasing.

Was this article helpful?