TL;DR: oMLX is an Apache 2.0 inference server for Apple Silicon that persists KV cache blocks to SSD, so a coding agent re-sending a long prefix waits 1–3 seconds instead of 30–90. It’s also the only Mac backend that genuinely batches concurrent requests. The catch: cold-start time-to-first-token runs 2–3× worse than plain mlx-lm, and the SSD cache does not let you run models bigger than your RAM.

	oMLX 0.6.2	Ollama (MLX backend)	mlx-lm (bare)
Best for	Coding agents, multi-user serving, long repeated contexts	Easiest setup, biggest model library	Fastest single request, scripting
Price	Free (Apache 2.0)	Free	Free
The catch	Cold TTFT 2–3× slower; macOS 15+ only	Serializes concurrent requests; MLX gains fade above ~35B	No cache persistence, no real server features

Honest take: If you point Claude Code, Cline, or any agent at a Mac and the same giant system prompt gets re-prefilled every turn, oMLX is the first tool that actually fixes the wait — switch. If you mostly chat with one model in one window, Ollama’s convenience still wins and the switch buys you nothing.

The pitch that put oMLX on GitHub’s trending page this week (just under 20,000 stars, 1.7k forks as of August 20, 2026) is one sentence long: SSD-backed KV cache cuts coding-agent time-to-first-token from 90 seconds to about 1. That claim comes straight from the Show HN thread and the author’s MLX discussion post, where users on long contexts report repeat-prefix TTFT dropping from 30–90 seconds to 1–3 seconds.

That is a real problem worth solving. Prefill is the tax every Mac pays on agentic workloads: an M-series GPU decodes tokens respectably, but chewing through a 30K-token repeated prefix before the first new token appears is where local coding agents on Apple Silicon go to die. We flagged the same bottleneck in our Ollama MLX guide — decode speed was never the whole story.

What oMLX actually is

oMLX (by GitHub user jundot) is an MLX-based inference server wrapped in a native SwiftUI menu bar app — not Electron — with a web admin dashboard at http://localhost:8000/admin. Requirements are strict: macOS 15 (Sequoia) or newer, any Apple Silicon chip (M1 through M4 family), Python 3.11–3.13 if you install from source. There is no Intel Mac path and no Windows/Linux path. License is Apache 2.0, so commercial use is clean.

It ships fast. The stable 0.6.2 release landed August 18, 2026, an 0.6.3 release candidate followed on August 19, and the releases page shows near-daily updates: 0.6.0 (August 16) added experimental distributed serving across multiple Macs, and 0.6.1 (August 17) added a dual-ANE/GPU prefill path for Qwen3.8 that the changelog measures at 18.9% faster prefill at 32K context, plus a 34% decode improvement at 16K from multi-token-prediction verification. Fast iteration cuts both ways — see the stability caveat at the end.

The server exposes OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /v1/models) and an Anthropic-compatible /v1/messages endpoint, which is why the README pitches it at “real coding work with tools like Claude Code.” It runs any mlx-lm-supported text model, plus vision models, OCR models, and embedding/reranker models, with tool calling and JSON-schema structured output.

The headline feature: a KV cache that survives on your SSD

Here is what the tiered cache actually does, per the project README:

• Hot tier (RAM): frequently accessed KV cache blocks stay in unified memory.
• Cold tier (SSD): when the hot cache fills, blocks are offloaded to disk as safetensors files.
• On the next request with a matching prefix — even after a server restart — blocks are restored from disk instead of recomputed.

Why this matters for agents specifically: a Claude Code or Cline session re-sends an enormous, mostly identical prefix (system prompt + tool definitions + conversation so far) on every single turn. Stock servers recompute that prefill each time, or keep it only in RAM until the next model swap or restart evicts it. oMLX treats the computed KV blocks as durable data. Restoring safetensors blocks from an NVMe SSD is enormously cheaper than recomputing attention over 30K tokens, which is how 30–90 seconds becomes 1–3.

Now the correction to the premise you may have seen in this week’s threads: SSD caching does not mean running models larger than your unified memory. The model weights still load into RAM in full — only the KV cache (the attention state for already-processed tokens) spills to disk. A 64GB Mac still cannot run a 70B model at Q8, oMLX or not. What the SSD tier buys you is effectively unlimited context memory across sessions, not extra model capacity. If you need bigger models than your Mac fits, that’s still a multi-tier hardware decision — or a rental problem (more below).

Disk usage is real but managed: the 0.6.0 release notes cite reworked linear SSD storage that cut one mixed-prefix workload’s cache footprint from 282.7GB to 15.3GB, and the cache directory is yours to place and cap.

Continuous batching: the only Mac backend that actually does it

The second feature is quieter but matters if anyone besides you talks to the server. An independent five-backend comparison (jaesolshin’s Apple Silicon backends test) found oMLX was the only backend that genuinely batches under concurrency: aggregate throughput scaled 1.29–1.40× going from 1 to 4 concurrent requests, while every other backend stayed flat. A separate M4 Max test measured near-linear scaling — 3.99× aggregate at 4 concurrent requests — with the machine in Low Power mode, where each request leaves GPU headroom for the others.

Ollama, by comparison, serializes requests on its Apple Silicon MLX runner — a known limitation tracked in ollama/ollama issue #17666. Two people (or two agent processes) hitting the same Ollama Mac take turns; on oMLX they overlap. Default is 8 concurrent requests, configurable per the README. If you’ve been eyeing vLLM for concurrency but your server is a Mac, this is the closest thing Apple Silicon has.

Setup: from zero to OpenAI-compatible endpoint

Install via Homebrew (or grab the DMG from releases — the menu bar app self-updates):

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx

# foreground, with SSD cache enabled at a path you choose:
omlx serve --model-dir ~/models \
  --paged-ssd-cache-dir ~/.omlx/cache \
  --hot-cache-max-size 20% \
  --max-concurrent-requests 8

The server comes up on port 8000; models dropped into --model-dir are auto-detected by type, or you download them from the admin dashboard at http://localhost:8000/admin. Any OpenAI client then works as a drop-in:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.6-27b-4bit", "messages": [{"role": "user", "content": "hello"}], "stream": true}'

Point a coding tool at http://localhost:8000/v1 (OpenAI-style) or use the /v1/messages endpoint for Anthropic-format clients. For wiring this into Cursor, Cline, or Claude Code as a local backend, our sister site’s guides at aicoderscope.com cover the client side; for the broader FOSS inference-server landscape, aifoss.dev tracks the ecosystem.

The speed reality: where oMLX loses

Raw single-request decode is not why you switch. Two independent tests agree on the shape of it:

• A Mac O’Clock benchmark found oMLX and bare mlx-lm essentially tied on mean decode speed, but oMLX was the most consistent engine tested (±0.4 tok/s run-to-run, vs ±2.3 tok/s for mlx-lm).
• A May 2026 M3 Max 64GB five-way comparison (mlx-lm / vllm-mlx / oMLX / Ollama / LM Studio, tested with Qwen3.6-35B-A3B across 2K–8K prompts) landed on the verdict: “for speed, mlx-lm; for API use, vllm-mlx; for functionality, oMLX; for ease of use, Ollama.” The same test found Ollama’s MLX path managed only ~30 tok/s on the 35B-class MoE where direct mlx-lm was roughly 3× faster — consistent with the pattern we documented in our Ollama v0.30 MLX review, where MLX’s advantage collapses as models grow.

The real problem you will hit, and the fix. Both tests flag the same trap: oMLX’s cold TTFT looks 2–3× worse than mlx-lm’s. The cause is structural — oMLX lazy-loads models, so your first request pays model-load-into-RAM plus tiered-cache bookkeeping plus the actual prefill, all stacked into one miserable first response. The fix is to fire one throwaway request right after starting the server (or pin the model from the admin dashboard) so the load cost is paid before anything you care about. Judge oMLX on request two, not request one.

One more number from the README worth calibrating: the “845 vs ~29 tok/s prefill on M3 Ultra” figure (roughly 30×) applies to GLM-5.2’s fused-DSA path with the optional custom Metal kernels installed. The standard install falls back to generic implementations. It’s a real capability, not the typical experience — treat it as a ceiling, not a baseline.

Which Mac makes oMLX worth it

oMLX doesn’t change silicon physics: decode speed still tracks memory bandwidth, and model ceiling still tracks unified memory. What it changes is how much of your hardware’s time gets wasted re-prefilling. Rough guidance, priced as of August 2026:

Machine	Unified memory	Street price (Aug 2026)	oMLX sweet spot
Mac Mini M4 Pro (64GB CTO)	64GB	~$1,999–$2,199	Headless agent server for 27–35B MoE models
Mac Studio M4 Max	36–128GB	from $2,499	Multi-user + big-context daily driver

The Mac Studio M4 Max base price sits at $2,499 after Apple’s June 2026 lineup-wide price increase (+$500 on this model); the 64GB Mac Mini M4 Pro configuration runs $1,999–$2,199 depending on GPU core count, per MacPrices tracking. Our Mac Mini M4 Pro review and Studio-vs-Mini comparison cover the bandwidth math in detail, and if you’re weighing a laptop instead, the M5 Max MacBook analysis applies unchanged — oMLX runs on anything M1 or newer with macOS 15.

No Mac at all, or need a model class no Mac fits? Renting a cloud GPU by the hour is the honest alternative for burst agent workloads — RunPod rents 24–96GB cards billed per hour, and our rent-vs-buy breakdown has the crossover math.

Verdict: who switches, who stays

Switch to oMLX if:

• You run coding agents (Claude Code, Cline, Cursor-with-local-backend) against a Mac. The SSD prefix cache attacks exactly your bottleneck, and the Anthropic-compatible endpoint removes a proxy layer.
• More than one person or process hits your server. Real batching (1.29–1.40× aggregate at 4 concurrent, vs flat for everyone else) is unique on this platform.
• You restart things a lot. Cache blocks surviving restarts is quietly the most practical feature here.

Stay on Ollama (or bare mlx-lm) if:

• You’re a single user chatting with a single model — you’ll pay oMLX’s cold-start tax and collect none of the batching or prefix-reuse wins.
• You want maximum single-shot speed for scripts: bare mlx-lm remains the ceiling.
• You value a battle-tested tool: oMLX shipped four releases in four days this week. That pace fixes bugs fast and introduces them fast. Run it for a week on your actual workload before making it a daily driver, and pin a version that behaves.

We’d also like to see more third-party benchmarks of the SSD restore path on slower external drives before calling the 1–3 second figure universal — every published number so far comes from internal NVMe.

FAQ

Does oMLX let me run models bigger than my Mac’s RAM? No. Only the KV cache (attention state) spills to SSD; model weights must still fit in unified memory. A 64GB Mac gains huge context reuse, not a bigger model ceiling.

Does it work with Claude Code or Cursor? Yes — it exposes both OpenAI-style /v1/chat/completions and Anthropic-style /v1/messages endpoints, so Anthropic-format clients connect without a translation proxy. Client-side setup guides live at aicoderscope.com.

What are the minimum requirements? macOS 15 (Sequoia) or newer on any Apple Silicon chip (M1–M4 family). No Intel, no Windows/Linux. Source installs need Python 3.11–3.13.

How much disk does the SSD cache eat? It scales with how much distinct context you process. The 0.6.0 release reworked storage so one mixed-prefix workload dropped from 282.7GB to 15.3GB on disk; you choose the directory with --paged-ssd-cache-dir and bound the RAM tier with --hot-cache-max-size.

Why is my first request so slow? Lazy loading — first request pays model load plus prefill together, which is why cold TTFT measures 2–3× worse than mlx-lm. Warm the model with a throwaway request after startup. If you’re seeing memory errors instead of slowness, that’s a different problem: see our MPS out-of-memory fix guide.

Recommended Gear

Products linked in this article:

• Mac Mini M4 Pro — 64GB CTO is the value headless oMLX server (~$1,999–$2,199, Aug 2026)
• Mac Studio M4 Max — from $2,499; the bandwidth and RAM headroom pick

Sources

• oMLX repository and README — GitHub (jundot/omlx)
• oMLX releases 0.6.0–0.6.3rc1 — GitHub
• Show HN: oMLX — SSD-backed KV cache cuts coding agent TTFT from 90s to 1s on Mac — Hacker News
• oMLX — MLX inference server with paged SSD caching, author’s post — ml-explore/mlx Discussion #3203
• Apple Silicon LLM Inference — Five Backends Compared — jaesolshin.com
• Local LLM inference on an M4 Max 128GB: Low vs High Power — kyu.co
• mlx-lm vs oMLX: What an Honest Benchmark Reveals — Mac O’Clock (Medium)
• mlx-lm / vllm-mlx / oMLX / Ollama / LM Studio compared on M3 Max 64GB — zephel01 (note.com)
• mlxrunner: requests are serialized — ollama/ollama issue #17666 — GitHub
• Mac Studio 2026: M4 Max vs M3 Ultra pricing — iTechGuides
• Apple Mac mini prices — MacPrices.net

Last updated August 20, 2026. Prices and specs change; verify current rates before purchasing. Some links are affiliate links — they cost you nothing and support the site.