TL;DR: Shard is an Apache 2.0 framework that splits an LLM into contiguous layer blocks across GPUs on separate machines and streams activations between them over the plain internet. Its headline receipt — GLM-5.2 (744B) at ~30 tok/s across six US states — is real, but it ran on six RTX PRO 6000 Blackwell workstation cards, roughly $48K–$56K of GPU. The genuinely consumer proof is gpt-oss-120B at ~40 tok/s on three RTX 4090s.

	Shard (pipeline, fixed cluster)	Mesh LLM (iroh P2P)	EXO (Apple-first pooling)
Best for	One model too big for any single box, dedicated node set	Routing requests to whichever box holds the model	Pooling Mac unified memory over Thunderbolt 5
Proven scale	744B over WAN, ~30 tok/s (6 nodes, 6 states)	Home-lab scale; sharding at ~1 node’s speed	DeepSeek 671B on 4× M3 Ultra, 32.5 tok/s
The catch	Every token crosses every node; spec decode is doing the heavy lifting	WAN routing adds RTT per token	NVIDIA path is a community fork

Honest take: Shard is the most impressive WAN inference demo yet published — and the receipts prove the engineering, not the economics. Unless six friends each own a 96GB card, treat it as a same-house multi-box tool, where the same design gets you the capacity without the 102ms ring tax.

A GitHub project called Shard made the rounds on Hacker News and r/LocalLLaMA this month with a claim that reads like a typo: GLM-5.2, the 744-billion-parameter open-weight frontier model, served at ~30 tokens per second across GPUs sitting in six different US states, connected by nothing fancier than the open internet. No RDMA, no Thunderbolt, no datacenter fabric.

We’ve covered distributed home inference twice before — EXO’s memory pooling and Mesh LLM’s P2P routing — and both times the honest verdict was “capacity, not speed.” Shard is the first project where the speed claim itself survives scrutiny. It’s worth understanding exactly how, because the trick is instructive even if you never join a six-state GPU ring — and because the “6 consumer GPUs” framing in the viral headline deserves a correction before you price out a cluster.

What Shard actually demonstrated (and on what hardware)

Shard’s repo publishes verifiable run receipts — JSON files with GPU UUIDs, node IPs, and output hashes — rather than just README numbers. Two matter:

• GLM-5.2, 744B (NVFP4, 78 layers) at ~30 tok/s sustained, greedy and deterministic, split 13 layers per node across six RTX PRO 6000 cards in Nevada, Texas, Minnesota, Missouri, and Utah, with a coordinator in Washington state. Receipt: glm52-nvfp4-wan-20260618.json (June 18, 2026).
• gpt-oss-120B (MXFP4, 36 layers) at ~40 tok/s (peak ~42) across three RTX 4090s in three different states. Receipt: gpt-oss-120b-wan-20260619.json (June 19, 2026).

You can pull the receipts yourself:

$ git clone https://github.com/leyten/shard && ls shard/docs/receipts/
fault-tolerance-20260623.json    glm52-nvfp4-wan-20260618.json
gpt-oss-120b-wan-20260619.json   sampling-lossless-20260623.json

Now the correction. The RTX PRO 6000 Blackwell is a 96GB workstation card that we profiled in June at a street price of roughly $8,000–$9,400. Six of them is $48,000–$56,000 in GPUs alone — before the six host machines under them. Calling that “consumer GPUs,” as the aggregated headlines did, stretches the word past breaking. The run that actually earns the label is the second one: three used 4090s at July 2026 street prices ($2,200–$2,350 each) is ~$6,600–$7,050 of GPU, which is expensive-hobbyist territory rather than fantasy.

The distinction matters because the two runs answer different questions. The 744B run proves the architecture scales to frontier weights over WAN. The 120B run proves a group of ordinary enthusiasts could replicate the idea this weekend.

How a token crosses six states: the pipeline mechanics

Pipeline parallelism is the oldest trick in multi-GPU inference: cut the model into contiguous blocks of layers, give each GPU one block, and pass the activations from block to block. Each node stores only its slice. For the GLM-5.2 run, that’s 13 of 78 layers per node — and since the full NVFP4 checkpoint is ~410GB (per the community NVFP4 model card, down from 1.5TB in BF16), each card holds roughly ~68GB of weights, leaving ~28GB of a 96GB card for KV cache. No single machine ever holds the model. The same arithmetic on gpt-oss-120B — a ~65GB MXFP4 checkpoint split three ways — puts ~22GB on each 24GB 4090, which is why that run works and why it’s tight.

The between-node payload is small — pipeline parallelism ships a per-token hidden-state vector, kilobytes not megabytes, the same property that makes Mesh LLM’s sharding workable. Notably, Shard’s docs do not publish a minimum bandwidth spec per node — we looked, because it’s the first question anyone building this should ask. What they publish instead is latency data, because latency, not bandwidth, is the enemy.

Here’s the problem in one number. Shard measured 22–75ms round-trip between its WAN nodes. A naive pipeline sends one token’s activations through all stages, gets one token out, and starts over — the full ring transit gates every single token. Shard’s ring, after optimization, completes in ~102ms. At one token per transit, that’s a hard ceiling around ~10 tok/s no matter how fast the GPUs are (our arithmetic, from Shard’s published ring latency).

Shard reports 30. Three things close the gap:

1. Speculative decoding across the swarm. The coordinator node holds no model layers at all — just the token embedding, the LM head, and a small CUDA-graphed GLM-4-9B draft model. The draft proposes a chunk of K tokens locally; the distributed 744B model verifies the whole chunk in one pass around the ring. When the draft is right, one 102ms transit yields several accepted tokens instead of one. The README puts it perfectly: speculative decoding is “marginal in a datacenter” but over WAN links it “becomes the whole game.” At ~30 tok/s measured, the swarm is netting roughly 3 accepted tokens per ring transit (again, our derivation).
2. Pipelined verification chunks. Multiple draft chunks are in flight through the ring simultaneously, so stages aren’t idle between verifications.
3. Ring topology with direct return. The tail node returns results straight to the coordinator — 7 hops around the ring rather than 12 hops relaying back through every stage. The team also found that where the coordinator sits matters enormously: repositioning it cut ring latency from 174ms to 102ms. That’s the actual problem-and-fix buried in the docs — if you ever build a multi-site pipeline, the coordinator belongs at the network centroid, not wherever your best GPU lives.

One more detail worth flagging as genuinely hard engineering: the outputs are greedy and deterministic, and a June 23 receipt documents lossless temperature/top-p/top-k sampling with speculative rejection at the tail. Distributed spec decode that provably matches single-box outputs is not a hack; it’s the correct algorithm.

The comparison that matters: Shard vs Mesh LLM vs EXO

Three distributed-inference projects have now crossed our desk in six weeks, and they solve different problems:

	Shard	Mesh LLM	EXO
Topology	Fixed ring, layers pre-assigned	Dynamic P2P, models live where they fit	Cluster pool, Apple-first
Model placement	Every node holds a slice of ONE model	Any node can hold a WHOLE model	Slices across pooled memory
WAN story	Designed for it (spec decode + pipelining)	Works, but RTT taxes every routed token	Thunderbolt 5 RDMA, same-desk
License	Apache 2.0	MIT	GPL-3.0
Headline number	744B @ 30 tok/s over WAN	~1 node’s speed when sharding	671B @ 32.5 tok/s on 4× M3 Ultra

The EXO number is the instructive contrast: four M3 Ultra Mac Studios linked by Thunderbolt 5 RDMA — Apple’s own numbers show TB5 cutting inter-device latency ~99% versus regular networking — deliver 32.5 tok/s on DeepSeek’s 671B, barely 1.5× a single node. Shard gets essentially the same throughput on a same-class model with its nodes a thousand miles apart. That’s the measure of how much work speculative decoding is doing.

Against Mesh LLM, the difference is philosophy. Mesh LLM assumes your models fit on some node and makes the network route to it; sharding is its fallback. Shard assumes the model fits on no node and makes the pipeline fast enough to tolerate. If your biggest ambition is 32B models across a house of mixed GPUs, Mesh LLM is the simpler tool. If your ambition is 120B+ open weights on cards you and two friends already own, Shard is the first framework built for exactly that.

Should you actually run this? The three honest scenarios

The six-state, 744B fantasy: no. Set aside the ~$50K of workstation cards. A single-box GLM-5.2 build already exists — jamesob’s four-RTX-PRO-6000 rig runs an expert-pruned 594B variant at ~80 tok/s with 460K context, 2.6× Shard’s WAN throughput, with no coordinator, no ring, and no five other people’s uptime to depend on. If you have workstation-card money, spend it on one chassis. (Or spend none: RunPod rents RTX PRO 6000s at about $1.99/hr — a weekend of experiments on 6× 96GB costs less than 1% of buying it.)

The three-4090 friend group: plausibly yes. This is the configuration Shard actually validated for normal people: ~$7K of used cards that three households might already own, producing 40 tok/s on a 120B model none of them could run alone. For context, gpt-oss-120B on a single RTX PRO 6000 does ~134 tok/s at 12K context (dropping to ~48 tok/s at full 131K, per Hardware Corner’s testing), and a $3,999 DGX Spark does ~38.5 tok/s. Three used 4090s over the internet matching a DGX Spark is a legitimately new option in the buy-vs-rent math. If you’re on the smaller sibling, our gpt-oss-20B guide covers the single-card path.

The same-house cluster: the sleeper use case. Everything that makes WAN inference hard — 22–75ms hops, coordinator placement, draft-model dependency — collapses when your nodes share a switch. On wired gigabit LAN, inter-node round-trips are sub-millisecond; the ring transit that costs Shard 102ms across six states costs microseconds across a rack. Two or three boxes with a used RTX 3090 each ($1,050–$1,450 on eBay, July 2026) become a 48–72GB pipeline for 70B–120B-class models at speeds where the network overhead is invisible. Shard’s roadmap (Phase 3: a permissionless swarm anyone can join with one command) is aimed at strangers pooling GPUs; the version of this that makes sense today is you pooling your own.

Two caveats before you wire anything up. First, the docs are explicit that a malicious node can reconstruct “a fraction of a user’s tokens” from the intermediate activations it processes — the wire protocol is encrypted (ChaCha20-Poly1305), but the nodes themselves see activations by design. Fine among friends; not fine on a permissionless swarm, whatever Phase 3 promises. Second, a pipeline is as reliable as its least-reliable node — there’s a fault-tolerance receipt dated June 23, 2026, but every node you add is another power flicker that can stall your cluster mid-generation.

FAQ

What internet speed do I need per node? Shard doesn’t publish a minimum bandwidth figure. The per-hop payload is a per-token hidden-state vector (kilobytes), so ordinary home upload speeds are unlikely to be the constraint — latency and jitter between nodes are. Until the project documents a spec, assume any stable wired connection qualifies and any Wi-Fi or CGNAT link is a risk.

Is Shard’s 30 tok/s claim trustworthy? It’s the best-documented claim in this space: receipts with GPU UUIDs, node IPs, and output hashes, plus greedy deterministic outputs that can be replayed against the model. We treat it as verified for that specific hardware and topology — six 96GB workstation cards and a tuned 102ms ring — not as a number your hardware will reproduce.

Can I mix different GPUs in one Shard pipeline? Layer blocks are sized to fit each node, and the receipts show homogeneous cards per run. Mixed VRAM should work in principle (smaller card, fewer layers), but the pipeline moves at the pace of its slowest stage — a 3060 in a ring of 4090s drags every token. The project’s ModelRuntime interface work is aimed at broader flexibility.

Shard vs Mesh LLM for a two-box home lab? If each box can hold the models you want, Mesh LLM — whole-model routing avoids the pipeline tax entirely. If your target model fits on neither box alone, Shard’s pipeline design is purpose-built for the split. On a LAN, either way, the network overhead is nearly invisible.

Does this replace renting a big cloud GPU? For bursty work, no — renting a 96GB card by the hour is still the cheapest way to touch 120B+ models occasionally. Shard changes the math for always-on access when the hardware is already owned across a group. Run the rent-vs-buy numbers with your actual duty cycle.

Sources

• leyten/shard — pipeline-parallel LLM inference across machines (GitHub README, receipts, roadmap)
• Shard README — benchmarks, ring topology, speculative decoding details
• GLM-5.2-NVFP4 model card (744B/40B MoE, 1.5TB→~410GB) — Hugging Face (lukealonso)
• Testing GPT-OSS 120B on RTX Pro 6000 Blackwell — Hardware Corner
• Optimizing GPT-OSS on NVIDIA DGX Spark — LMSYS Blog
• jamesob/local-llm — the 4× RTX PRO 6000 GLM-5.2 build guide (GitHub)
• 1.5 TB of VRAM on Mac Studio: RDMA over Thunderbolt 5 — Jeff Geerling
• RTX PRO 6000 Blackwell 96GB — Newegg listing
• RTX PRO 6000 dropped to $7,999 — VideoCardz
• RTX 3090 used price tracker, July 2026 — BestValueGPU
• Mesh-LLM/mesh-llm — GitHub repository
• RTX Pro 6000 GPU rental pricing — RunPod

Last updated July 28, 2026. Prices and specs change; verify current rates before purchasing.

For the FOSS-license angle and GLM-5.2 self-hosting context, aifoss.dev covers the Apache 2.0 internals; for wiring a distributed endpoint into your coding agent, see aicoderscope.com.

Recommended Gear

• RTX 4090 — 24GB; three of them ($2,200–$2,350 each used, July 2026) reproduce Shard’s verified consumer-grade 120B run.
• RTX 3090 — 24GB at $1,050–$1,450 used; the budget node for a same-house pipeline.
• RTX PRO 6000 Blackwell — 96GB; the card the 744B receipt actually ran on, ~$8,000–$9,400 street.