TL;DR: AirLLM genuinely runs Llama 3.3 70B on a 4GB GPU by streaming one transformer layer at a time — no quantization required. The cost is that every generated token re-reads the model from storage, so a fast NVMe drive bottoms out near 20 seconds per token at full precision. It’s a capability demo and a batch tool, not a chat replacement.

	AirLLM on your 4GB card	llama.cpp on 4GB GPU + 64GB RAM	Used RTX 3090 (24GB)
Best for	Proving 70B runs at all; overnight batch jobs	Actually usable 70B at home on a budget	Daily-driver local AI
70B speed	~20 s/token cold at fp16; seconds/token compressed	2–4 tok/s (Q4_K_M in RAM)	8–15 tok/s (70B partial offload), ~95 tok/s on 7B
Cost	$0 if you own the card	~$100–200 of RAM	~$1,050–$1,299 used, Aug 2026
The catch	Disk is the engine; GPU mostly waits	Needs the RAM, not the GPU	Price up ~25% since March

Honest take: AirLLM is one of the most clever party tricks in local AI, and you should try it once — then buy RAM or a 24GB card for actual work. If your machine has enough system memory to make AirLLM tolerable, llama.cpp with a quantized model in that same RAM is 10–50× faster.

AirLLM hit the Hacker News front page again this week, years of GPU pricing frustration compressed into one headline: “70B inference with single 4GB GPU.” The repo sits at roughly 32,000 stars as of August 2026, ships version 3.2.0 on PyPI (released August 18, 2026), and the claim on the tin is accurate. A GTX 1650, an RTX 3050 laptop chip, the 4GB slice of VRAM your iGPU borrows — all of them can produce tokens from Llama 3.3 70B at full fp16 precision, no quantization, no pruning, no distillation.

What the headline doesn’t say is what your SSD will be doing while that happens. This guide covers the mechanism, the real speed numbers nobody puts in the README, and the decision math against the two obvious alternatives: more system RAM, or a used RTX 3090.

The trick: your GPU only ever holds one layer

Llama 3.3 70B is 80 transformer layers with an 8,192-wide hidden state. At fp16, the full weight set is 141.12GB — that’s the F16 file size on Bartowski’s GGUF ladder, and it’s why “just load it” requires five RTX 5090s’ worth of VRAM or a data-center card.

AirLLM’s observation: inference is sequential. Layer 1’s output feeds layer 2, layer 2’s feeds layer 3, and at any given instant the GPU is only doing math on one layer. So AirLLM loads layer 1 from disk to VRAM, computes, discards it, loads layer 2, and walks the whole stack that way. Peak VRAM is one layer’s weights (141GB ÷ 80 layers ≈ 1.75GB average) plus activations and the KV cache — which is how the project’s compatibility table lists a 70B at ~4GB, Llama 3.1 405B at ~8GB, and DeepSeek-V3 671B at ~12GB.

The library has kept pace with the 2026 model churn: v3.0 (June 2026) added FP8 support, and an August 2026 update added Qwen3.8-27B, which the maintainer measured end-to-end at 3.33GB of VRAM on an RTX 3090. Llama 2 through 4, Qwen through 3.8, DeepSeek V2/V3/R1, Mistral, Mixtral, Phi, Gemma, and Kimi K3 are all supported through one AutoModel interface, with MLX backing on Apple Silicon. License is Apache 2.0 — no strings.

Setup is genuinely three lines:

pip install airllm

from airllm import AutoModel

model = AutoModel.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
input_ids = model.tokenizer(["What is the capital of Australia?"],
                            return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=20)
print(model.tokenizer.decode(output[0]))

What you should see, eventually — a normal, full-precision answer, with nvidia-smi never rising above ~4GB while it generates:

The capital of Australia is Canberra.

The first run downloads the full-precision weights — budget 140GB+ of free disk before you start, which on a machine with a 4GB GPU is often the second surprise.

Then you wait. And this is where the interesting part starts.

The physics: every token reads the entire model off disk

Here’s the problem the layerwise trick cannot engineer away. Autoregressive generation runs the full forward pass — all 80 layers — once per token. Your 4GB card can’t keep any meaningful fraction of 141GB resident, so every one of those layers has to come back across the PCIe bus from storage, for every single token.

Do the division. A fast Gen4 NVMe drive like the Samsung 990 Pro sustains about 7GB/s sequential reads. 141GB of fp16 weights ÷ 7GB/s ≈ 20 seconds per token, before any compute happens (Umesh Malik’s writeup walks the same math). That’s the floor on an uncached full-precision run with a top-tier consumer SSD. A 500-token answer at that rate is just under three hours. On a SATA SSD it’s 4× worse; on a spinning drive, community reports put it below 0.1 tokens per second — a number that stops being inference and starts being geology.

The GPU, meanwhile, is nearly idle. As the DEV Community analysis put it, AirLLM turns your SSD into the inference engine — sequential read speed is the spec that determines your tokens/sec, not CUDA cores. It’s the most extreme case of the rule that governs every article on this site: LLM decode speed is memory bandwidth, full stop. A used RTX 3090 moves weights at 936 GB/s. A Gen4 NVMe moves them at 7 GB/s. That 134× gap is the entire story.

Two settings claw some of it back:

• compression='4bit' — block-wise quantization (needs bitsandbytes) shrinks the read to roughly a quarter, and the repo claims up to 3× faster inference. Note this abandons the “no quantization” purity that makes the headline impressive; you’re now running a ~35GB 4-bit model very slowly instead of a 141GB fp16 model extremely slowly.
• Prefetching — on by default, overlaps the next layer’s disk read with the current layer’s compute. The repo credits it with about 10%.

model = AutoModel.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    compression='4bit'   # ~35GB read per token instead of ~141GB
)

What people actually measure

Numbers from hands-on runs, not the README:

Setup	Measured speed	Source
70B, 4-bit compression, warm page cache	24 tokens in 80.9s (~0.3 tok/s)	QWE AI Academy hands-on
70B, decent NVMe, community range	~5–35 s/token	Starlog analysis
70B fp16, 7GB/s Gen4 NVMe, cold	~20 s/token floor (bandwidth math)	Umesh Malik
Kimi K3 (1T-class) on RTX 6000 Ada 48GB, v3.1.0	292 s/token	El Solitario test

Two things in that table deserve a closer look.

The 0.3 tok/s best case is only possible because of the OS page cache. With 4-bit compression the working set drops to ~35GB — and if your machine has more system RAM than that, Linux quietly keeps the layers cached in memory after the first pass. The “disk” reads become RAM reads and the speed triples or better. Which reveals the trap: AirLLM is only tolerable on machines with enough RAM that you didn’t need AirLLM. If you have 64GB of system RAM, llama.cpp will hold the entire Q4_K_M GGUF (42.5GB) resident and decode at 2–4 tok/s on CPU alone — 10–50× faster than AirLLM’s honest cold-cache rate, on the same hardware. Our 70B-on-24GB guide has the full offload ladder.

The Kimi K3 row is the cautionary tale at the other end: layer streaming on a $7,000 48GB workstation card still delivered nearly five minutes per token, because the model is a 1.56TB monster and the disk is still the engine. VRAM headroom doesn’t rescue the architecture — bandwidth does.

One aside on hardware anxiety: streaming 141GB per token sounds like it should murder your SSD, but endurance ratings (TBW — terabytes written) count writes, not reads. AirLLM’s read-heavy pattern is boring, safe, sequential traffic. The wear happens once, when you download the weights. If you’re picking a drive for this kind of abuse anyway, our NVMe-for-local-AI guide covers what actually matters.

The decision: AirLLM vs quantization vs real hardware

The question the Hacker News thread kept circling: if you have a 4GB GPU and 70B ambitions, what’s the actual best move? Ranked by dollars:

$0 — AirLLM, eyes open. Overnight batch jobs are the legitimate use case: dataset labeling, offline evaluation runs, generating a few hundred fp16-exact completions where quantization noise would contaminate the comparison. Queue prompts before bed, collect them at breakfast. It is also the only way to touch a 405B or 671B model on consumer hardware at all, and there’s real value in verifying a giant model’s output on your own machine before renting serious compute.

~$100–200 — RAM, the unglamorous winner. 64GB of DDR4/DDR5 plus llama.cpp turns the same 4GB-GPU machine into a 2–4 tok/s 70B box (Q4_K_M resident in RAM, a handful of layers offloaded to the card). Slow, but two orders of magnitude better than AirLLM cold, and it makes every smaller model comfortable too. Check our system RAM guide for the sizing math.

~$1,050–1,299 — the used RTX 3090. Market average $1,264 across 366 listings in August 2026, with eBay floors around $1,050 — up from ~$1,010 in March, so the trend isn’t your friend. 24GB runs a 70B via partial offload at 8–15 tok/s (even Q2_K’s 26.38GB doesn’t fully fit, so some layers ride in RAM), runs the 27B–35B class at full speed, and does ~95 tok/s on 7B models. This is the card the whole buying guide keeps landing on for a reason.

~$1.39/hour — rent the answer. A RunPod A100 80GB (price as of July 2026) holds a 70B Q4 fully resident with room to spare. If your 70B need is occasional, renting beats owning — an evening of experimentation costs less than lunch, at 50–100× AirLLM’s speed.

The honest framing: AirLLM’s 70B-on-4GB is real in the way a bicycle crossing the Sahara is real. It happened, it proves something interesting about the route, and you should still book the flight.

There’s one more audience worth naming — developers. If you’re wiring a local model into Cursor, Cline, or another coding agent, AirLLM’s latency profile makes it a non-starter; agent loops need dozens of fast round-trips, and 20-second tokens break every timeout in the stack. Use a model that fits your VRAM instead — the 6GB tier list is the realistic menu for small cards. For the FOSS angle on AirLLM’s internals and license, our sister site aifoss.dev covers self-hosted tooling in depth.

FAQ

Does AirLLM quantize the model? Not by default — that’s the headline feature. Layers run at full fp16 precision, so outputs are bit-identical to what a data-center deployment produces. The optional compression='4bit'/'8bit' flag adds block-wise quantization for roughly 3× speed, at which point you’ve rejoined the quantization world you were trying to avoid.

Will it work on my 4GB laptop GPU? If it runs CUDA (or MLX on Apple Silicon) and you have 140GB+ of free disk for the fp16 weights, yes. Speed will track your SSD’s sequential read rate almost linearly — a laptop with a Gen3 NVMe (~3.5GB/s) will roughly double the per-token times quoted here.

Why is my second run faster than my first? OS page cache. Whatever fraction of the layers fits in free system RAM gets served from memory on subsequent passes. This is also why speed reports vary so wildly between machines with 16GB and 128GB of RAM running the “same” 4GB GPU.

Is AirLLM faster than llama.cpp CPU offloading? No — not on any machine we’ve seen numbers for. llama.cpp reads weights from system RAM (~50–80GB/s on dual-channel DDR5); AirLLM cold-reads from disk (~7GB/s best case). If the model fits in your RAM at some quantization, llama.cpp wins by an order of magnitude. AirLLM’s advantage is capacity, not speed: it works when the model doesn’t fit in RAM either.

What’s it actually good for? Overnight batch generation, fp16-exact offline evaluation, demoing that a 70B/405B runs at all on minimal hardware, and inspecting a frontier-sized open model’s behavior before committing to rented GPUs. Interactive chat is not on the list.

Recommended Gear

• Used RTX 3090 24GB — the perpetual value answer for real local 70B-adjacent work; ~$1,050–$1,299 used as of August 2026
• Samsung 990 Pro 2TB — if the disk is going to be your inference engine, it had better be a 7GB/s one

Sources

• AirLLM repository — GitHub (lyogavin/airllm)
• airllm 3.2.0 — PyPI
• AirLLM 70B inference with single 4GB GPU — Hacker News discussion
• Llama-3.3-70B-Instruct-GGUF file sizes — Bartowski, Hugging Face
• Llama 3 70B architecture (80 layers, 8192 hidden) — EmergentMind
• AirLLM hands-on: 24 tokens in 80.9 seconds — QWE AI Academy
• Run 70B LLM on 4GB GPU: AirLLM’s real tradeoff (20s/token floor math) — Umesh Malik
• How AirLLM trades speed for accessibility — Starlog
• AirLLM runs a 70B model on a 4GB GPU — DEV Community
• AirLLM with Kimi K3: 292 seconds per token — El Solitario
• RTX 3090 price tracker, August 2026 — BestValueGPU

Last updated August 21, 2026. Prices and specs change; verify current rates before purchasing.