TL;DR: Mesh LLM stitches whatever GPUs you own — in different rooms or even different cities — into a single OpenAI-compatible endpoint over iroh’s peer-to-peer network. It’s genuinely clever for NVIDIA-first Linux home labs and needs no RDMA or Thunderbolt. But peer-routed and sharded requests pay a network-latency tax, so it buys you capacity, not speed.

	Mesh LLM (iroh P2P)	EXO (Apple-first)	RDMA Mac cluster (TB5)
Best for	Mixed NVIDIA/Linux GPUs on any network, including WAN	Pooling Apple Silicon unified memory	Same-room Mac-only clusters chasing throughput
Interconnect	iroh QUIC over any link (LAN, Wi-Fi, internet)	Thunderbolt/LAN, MLX backend	Thunderbolt 5 RDMA, ~80 Gb/s, same LAN
NVIDIA support	First-class (llama.cpp backend)	Community exo-cuda fork only	N/A
The catch	Peer/sharded requests add network RTT	GPU path strongest on macOS, not NVIDIA	Mac-only, expensive, single-room

Honest take: If you’re a Linux home-labber with two or three NVIDIA boxes and you keep hitting the VRAM ceiling, Mesh LLM is the most flexible way yet to pool them — just run the big model where the GPU lives and treat cross-machine routing as cold-start overflow, not your interactive chat path.

---

What Mesh LLM actually is

Mesh LLM is an open-source (MIT-licensed) project that pools the GPUs and memory of several machines and exposes the whole thing as a single OpenAI-compatible API at http://localhost:9337/v1. Point any standard OpenAI client — Open WebUI, Continue, a curl script — at that endpoint, and it talks to your mesh without modification.

It’s built in Rust on three load-bearing pieces: iroh for the networking, llama.cpp for the actual inference, and a scheduler the project calls Skippy for deciding where each request runs. The catalog ships with 40+ models spanning from half-a-billion-parameter models that fit on a laptop up to 235B mixture-of-experts giants, with reviewed llama.cpp parity covering 72 P0/P1 family rows and 89 certified rows total across Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi, Granite, Hunyuan, EXAONE, Cohere, Falcon, and RWKV.

The pitch that put it on the Hacker News front page (179 points, 41 comments on July 11, 2026) is simple: you’re “one GPU short” of the model you want, but you own two machines. Instead of buying a bigger card in a market where a used RTX 3090 now runs $1,050–$1,450, you network the boxes you already have and run the model across both.

Why iroh is the interesting part

The reason to care about Mesh LLM over the alternatives isn’t the inference — that’s llama.cpp, same as Ollama or LM Studio. It’s the transport.

iroh (from n0) is a peer-to-peer networking library — think Tailscale or Steam’s networking, but as a library you embed in your app rather than a service you install. It gives you authenticated, NAT-traversing QUIC connections between any two machines, addressed by public key. In practice that means streaming a token’s activations to the next machine in a pipeline is the same primitive as talking to localhost — the library hole-punches through home routers and, when it can’t, falls back to relays automatically.

That’s what frees Mesh LLM from the constraints that box in its rivals. EXO’s fast path assumes machines on the same LAN. Apple’s RDMA-over-Thunderbolt-5 Mac clusters assume machines in the same room, cabled together. iroh assumes nothing: two RTX 3090 boxes in different rooms on the same Wi-Fi, or a friend’s spare GPU across the city, are both just peers. Mid-2026 iroh builds even added QUIC congestion-control tuning for high-latency Wi-Fi links, which fixed the mid-inference stalls earlier builds suffered on flaky home networks.

The three ways a request gets served

Skippy decides how to handle each incoming request, and understanding these three modes is the whole game — because two of them are fast and one of them isn’t:

1. Local. If the node you hit can hold the whole model, it serves it locally with no cross-machine traffic at all. This is full-speed llama.cpp — identical to running Ollama on that box.
2. Peer routing. If a different peer already has the requested model loaded, the request is routed to that peer by its model field. The peer runs the whole model on its own GPU; you just pay one network round-trip to ship the prompt over and stream tokens back.
3. Sharded pipeline. If no single node can hold the model, Skippy splits it into stages by layer ranges and runs it as a pipeline — several modest machines each holding a slice, none of them able to run it alone.

Modes 1 and 2 are where Mesh LLM shines. Mode 3 is where you need to be honest with yourself, and it’s the same lesson we covered with EXO: pipeline sharding adds capacity, not speed.

The latency reality nobody puts in the headline

Here’s the physics the marketing skips. In a sharded pipeline, a token flows through stage 1’s layers, gets shipped over the network to stage 2, through those layers, and so on. For any single request, only one node is doing work at a time — the others wait their turn. You get the combined memory of every node, but roughly the throughput of one node, minus whatever the network adds between stages.

The good news is the between-stage payload is small. Pipeline parallelism ships a hidden-state vector per token across each stage boundary — kilobytes, not megabytes — so this is a latency problem, not a bandwidth one. On a wired gigabit LAN, inter-node round-trips are sub-millisecond and the overhead is nearly invisible. Over Wi-Fi it’s a few milliseconds. Across a WAN to a friend’s GPU, you’re adding tens of milliseconds of real-world RTT per hop, and if iroh can’t hole-punch and falls back to a relay, more.

So the practical rules that fall out of this:

• Local requests run at native single-GPU speed. As our VRAM tier guide documents, a used RTX 3090 does roughly 90–95 tok/s on a 7B Q4 model and around 40–50 tok/s on a 22B dense model — Mesh LLM changes none of that when the model fits locally.
• LAN peer routing is the sweet spot: near-native speed plus a sub-millisecond routing hop. A 24GB box serving a 27B Q4_K_M to the rest of your LAN is a great use of Mesh LLM.
• WAN peer routing only makes sense for cold-start offload — “my box is busy, borrow the model on the box across town” — not for interactive chat, where the added RTT is felt on every token.
• Sharded pipelines get you into models you otherwise couldn’t load at all (two 24GB nodes = a 48GB working budget for a ~40B-class Q4 model), but expect throughput in the neighborhood of a single node, not double.

What hardware you actually need

There’s no special interconnect requirement — that’s the point. Any GPU that llama.cpp supports works: NVIDIA (CUDA), AMD (ROCm/Vulkan), Apple Silicon (Metal), even CPU-only nodes for small models. What matters is per-node VRAM, because that ceiling decides which model shards a node can hold.

A rough map for the common home-lab tiers:

Per-node VRAM	Runs locally (Q4_K_M)	Role in a mesh
8–12 GB (RTX 3060/4060)	up to ~8B–12B	Serves small models to peers; a pipeline stage
16 GB (RTX 5060 Ti)	up to ~14B, tight 22B	Solid single-model server node
24 GB (RTX 3090 / RTX 4090)	27B–32B comfortably	The workhorse node; pair two for a ~48GB pool
32 GB (RTX 5090)	32B with long context	Best single-node member

Two 24GB nodes give you a 48GB virtual pool — enough for a ~40B-class model in Q4 via sharding, or, far better, enough to keep two different 24GB-class models hot on the mesh and route between them. That second pattern is the one to reach for: it stays in fast peer-routing mode and never pays the pipeline tax. And before you build a multi-box mesh, size the power: two 3090s pulling ~350W each want real headroom, which our PSU sizing guide walks through.

Setup walkthrough

Getting a two-machine mesh running is genuinely quick. On each machine:

# 1. Install (both machines)
curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash

# 2. First-run setup — detects GPU, pulls a starter model
mesh-llm setup

On the first machine, start serving. On the second, join the mesh with the invite token the first node prints:

# Machine A (the one you started first) prints a join token
# Machine B joins it:
mesh-llm serve --join <token>

Because iroh handles NAT traversal, the second machine doesn’t need to be on the same subnet or have any ports forwarded — the invite token carries what iroh needs to establish the direct QUIC connection (see docs/MESHES.md in the repo for the full ticket flow).

Verify both nodes’ models are visible from the unified endpoint:

curl -s http://localhost:9337/v1/models | jq '.data[].id'

Then run a completion against any model in the mesh — Skippy routes it to whichever node can serve it:

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

If that model lives on the other box, you’ve just confirmed peer routing works — the request left your machine, ran on the peer’s GPU, and streamed back.

Mesh LLM vs EXO vs an RDMA Mac cluster

These three tools solve overlapping problems from different starting points, and picking wrong wastes money.

• EXO is Apple-Silicon-first. Its strongest GPU backend is MLX on Macs; NVIDIA support lives in the community exo-cuda tinygrad fork and runs behind native CUDA. If your home lab is Mac Studios, EXO’s MLX throughput is hard to beat. If it’s Linux NVIDIA boxes, EXO is the awkward fit.
• RDMA over Thunderbolt 5 is the throughput king — but Mac-only, cabled, same-room, and expensive. It exists to make a stack of M-series Macs behave like one giant machine for trillion-parameter models. It is not an option for a mixed or NVIDIA lab.
• Mesh LLM wins precisely where the other two don’t: mixed-vendor GPUs, Linux-first, and any network topology including the open internet. It trades peak throughput for flexibility, and for most home labs that already own a couple of NVIDIA cards, that’s the right trade.

The choice is basically: Mac-only and chasing raw speed in one room → RDMA cluster. Apple Silicon and pooling unified memory → EXO. Everything else, especially NVIDIA on Linux across rooms or houses → Mesh LLM.

What if you don’t have a second GPU box?

Mesh LLM only helps if you have machines to mesh. If you’re a single-GPU household and you occasionally need to run something that doesn’t fit — a 70B model for a one-off batch job, say — spinning up a cloud GPU by the hour is cheaper than buying a second card in the current memory-shortage market. A RunPod instance for a few hours costs a fraction of a used 3090, and you can even make that rented box a temporary peer in your mesh for the duration of the job, then tear it down. That’s a far better use of cloud money than paying a monthly API bill for a model you only touch occasionally.

Honest take

Mesh LLM is the first distributed-inference project that feels designed for the home lab it’s aimed at rather than adapted from a data-center assumption. iroh’s P2P transport removes the same-room, same-vendor constraints that make EXO and RDMA clusters niche, and the OpenAI-compatible endpoint means your existing tools just work. The MIT license and llama.cpp foundation mean there’s nothing proprietary to get locked into.

But treat it as a capacity and convenience tool, not a speed multiplier. The winning pattern is keeping big models hot on the 24GB box where the GPU actually lives and letting the rest of the house route to it — fast peer routing, no pipeline tax. Reach for sharded pipelines only when a model genuinely won’t fit any single node, and go in knowing you’ll get one node’s throughput with several nodes’ memory. Do that, and it’s the most flexible way to squeeze real value out of the GPUs already sitting in your home lab. If you want the FOSS-license and architecture deep-dive, aifoss.dev is covering the MIT-licensed internals; for the coding-agent angle on wiring a local endpoint into your editor, see aicoderscope.com.

FAQ

Does Mesh LLM make inference faster? Only when it keeps you in local or LAN-peer mode. Running a model that fits on one node is full native speed. Sharding a too-big model across nodes gives you the memory of all of them but roughly the throughput of one — distributed pipelines add capacity, not speed.

Do I need Thunderbolt or RDMA like the Mac clusters? No. That’s the headline feature. iroh runs over any network — gigabit Ethernet, Wi-Fi, or the open internet — using QUIC with automatic NAT traversal. No special cabling or interconnect hardware.

Does it work with NVIDIA GPUs on Linux? Yes, natively — inference runs on llama.cpp, so any CUDA GPU works out of the box. This is the main advantage over EXO, whose NVIDIA support is a community fork.

Can I mesh a machine across the internet? Yes. iroh addresses peers by public key and hole-punches through NATs, so a peer across town is just another node. But expect network RTT on every token for WAN-routed requests — use it for cold-start offload, not interactive chat.

What’s the minimum VRAM per node? Whatever holds the model you want that node to serve. An 8GB card handles models up to ~8B; a 24GB card comfortably runs 27B–32B in Q4. Per-node VRAM decides which shards a node can hold in a pipeline.

Is it free? Yes — MIT licensed, built with Rust, iroh, and llama.cpp. No cloud account, no subscription.

Sources

• Mesh LLM: distributed AI computing on iroh — iroh.computer blog
• Mesh-LLM/mesh-llm — GitHub repository (README, install, API, license)
• Mesh LLM: distributed AI computing on iroh — Hacker News discussion (179 pts, Jul 11 2026)
• iroh — peer-to-peer QUIC networking library (n0)
• Mesh-LLM llama.cpp parity / model catalog — mesh-llm docs
• RTX 3090 used price tracker, July 2026 — BestValueGPU

Recommended Gear

• RTX 3090 — 24GB, the value workhorse node for a mesh; $1,050–$1,450 used (July 2026).
• RTX 5060 Ti — 16GB, an affordable single-model server node.
• RTX 4090 — 24GB, faster workhorse for the primary node.
• RTX 5090 — 32GB, the best single-node member for long-context 32B models.

Prices as of July 2026 and move weekly in the ongoing memory shortage — verify at the retailer before buying.