Running a 28.9M-Parameter LLM on an $8 Microcontroller: What Tiny Edge AI Means for Home Lab Builders in 2026

By RunAIHome Team · 14 min read
edge-aiesp32tiny-llmlocal-llmhome-labper-layer-embeddings

TL;DR: A developer got a 28.9M-parameter LLM generating text at ~9.5 tokens/second on an $8 ESP32-S3 — fully on-chip, no cloud — by parking 25M of those parameters in flash and reading only ~450 bytes per token. It writes TinyStories-grade prose, not answers; it will not replace anything in your rack. The memory trick, borrowed from Google’s Gemma line, is the part that matters.

ESP32-S3 (this project)Raspberry Pi 5Used RTX 3090 rig
Best forAlways-on edge node: classification, toy generationSmall-model inference (sub-1B)Real LLM work (7B–70B quants)
Hardware cost~$8 board$80 (8GB MSRP)$1,050–$1,306 for the card alone (eBay, July 2026)
Power, 24/7 for a year~3.5 kWh ≈ $0.42 at $0.12/kWhtens of kWh350+ kWh idle alone (~40W floor)
The catch28.9M params = short stories and simple patterns onlyCan’t touch 7B-class qualityIdle draw and price; overkill for sensor-grade tasks

Honest take: Don’t buy an ESP32-S3 expecting a pocket ChatGPT — buy one if you want an always-on, sub-half-watt AI sensor node that never phones home, and keep the real model on your GPU server. The two-tier pattern is the actual takeaway here.

On July 25, 2026, a project called esp32-ai hit the Hacker News front page: a 28.9-million-parameter language model running entirely on an ESP32-S3 development board — the ~$8 Wi-Fi microcontroller that powers half the smart-home gadgets in your house — writing short stories to a little screen at about 9.5 tokens per second. Nothing leaves the chip. No API key, no server, no subscription.

The reflexive response is “cute, but useless.” That’s half right. The model itself is a toy. The engineering underneath it is not, and it points at something home lab builders should actually plan around in 2026: a memory-hierarchy trick that lets models bigger than your fast memory run anyway — the same idea, at wildly different scale, that Google ships in Gemma and that your GPU server already exploits every time you offload MoE experts to system RAM.

What actually shipped: 9.5 tok/s from a chip that costs less than lunch

The project is esp32-ai by slvDev, MIT-licensed, with training code, quantization scripts, and firmware in the repo. The headline numbers, straight from the project:

SpecValue
Model size28.9M parameters (25M of them in a flash lookup table)
Quantization4-bit → 14.9MB total on disk
HardwareESP32-S3: dual-core LX7 @ 240MHz, 512KB SRAM, 8MB PSRAM, 16MB flash
Speed~9.5 tok/s end-to-end (9.7 tok/s pure compute)
Flash reads per token~450 bytes (~6 embedding rows)
Training dataTinyStories (synthetic short stories)
LicenseMIT

For scale: the previous well-known result on this class of hardware was Dave Bennett’s 2024 esp32-llm, which ran a 260K-parameter TinyStories checkpoint from Karpathy’s llama2.c lineage at 19.13 tok/s, using both cores and the ESP-DSP dot-product routines. That model was about 1MB. This one is 111× larger in parameter count and still generates at a readable pace on the same $8 silicon. That jump did not come from the chip getting faster — it came from rethinking where the parameters live.

The trick: 25 million parameters that never touch RAM

Here’s the problem that made Bennett’s 2024 approach a dead end at larger sizes. Autoregressive generation reads essentially the whole model once per token. A 260K model (~1MB) fits in PSRAM, and reading 1MB per token from PSRAM is cheap — hence 19 tok/s. Scale the model up 100× and every token now has to pull 100× more bytes through a memory bus that was slow to begin with. The naive path to 28.9M parameters on this chip is single-digit seconds per token, not single-digit tokens per second.

The fix borrows Google’s per-layer embeddings (PLE) design from the Gemma family. In Gemma 3n, PLE is what lets the E2B model carry 5B total parameters while loading only about 1.91B into accelerator memory — the per-layer embedding tables sit in slower, cheaper memory (CPU RAM on a phone or PC) and only the rows the current token actually needs get fetched. Google built it so phones could run bigger models than their VRAM-equivalent allows.

esp32-ai applies the same split three rungs down the memory ladder:

Memory tierSizeWhat lives there
SRAM (fast)512KBThe dense transformer core (~560K params) doing the actual reasoning
PSRAM (medium)8MBOutput head and working buffers
Flash, memory-mapped (slow, huge)16MBThe 25M-parameter embedding lookup table

The insight that makes it work: an embedding table is a lookup, not a matrix multiply. You never need the whole table — you need a handful of rows per token. So the table sits in flash, memory-mapped (XIP), and each token triggers roughly six row reads totaling ~450 bytes. Flash is glacially slow for bulk reads, but 450 bytes is nothing. Meanwhile the small dense core that runs every token stays in SRAM where it belongs. Result: 86% of the model’s parameters never occupy RAM at all, and generation speed is set by the 560K-parameter core, not the 28.9M-parameter total.

If that pattern sounds familiar, it should — it’s the microcontroller version of what you’re already doing when you run a MoE model with expert offload in llama.cpp, or when Gemma 4 QAT checkpoints squeeze a 26B into 15GB. The 2026 theme across every scale of local AI, from $8 chips to $50K quad-GPU builds: stop asking “does the model fit in fast memory?” and start asking “what does each token actually need to read, and from where?”

Where 28.9M parameters sits on the ladder

Be clear-eyed about what a 28.9M-parameter model is. It is not a small ChatGPT. It was trained on TinyStories — synthetic children’s stories with a deliberately tiny vocabulary — and short coherent stories are all it writes. No questions, no instructions, no facts, no code. Ask nothing of it.

ModelParamsSize on diskHardwareSpeedWhat it can actually do
esp32-llm (2024)260K~1MBESP32-S3, 2MB PSRAM19.13 tok/sBarely-coherent story fragments
esp32-ai (this project)28.9M14.9MB (4-bit)ESP32-S3, $8~9.5 tok/sCoherent short stories; simple pattern tasks
LFM2.5-230M230Munder 1GB RAMRaspberry Pi 542 tok/sReal classification, extraction, short summaries
Gemma 3n E2B5B total / 1.91B effective~2GB-classPhones, laptopsvariesGenuine instruction-following, multimodal
Llama-class 8B at Q4_K_M8B~5GBUsed RTX 309080–100 tok/sDaily-driver chat, coding help, RAG

Each rung is roughly an order of magnitude in capability, and the jumps are not linear — the difference between 28.9M and 230M is bigger in practice than the parameter ratio suggests, and the jump from 230M to 8B is bigger still. We covered the Pi 5 rung in detail in our LFM2.5 on Raspberry Pi 5 guide, and the GPU rungs in best local AI models by VRAM tier.

What fits in 28.9M parameters, realistically: task-specific micro-models. Wake-word and intent detection, sentiment flags, anomaly classification on sensor streams, simple template-ish generation. Purpose-trained on one narrow job, a model this size can be genuinely useful — that’s the entire premise of the TinyML world, which was doing keyword-spotting on microcontrollers years before anyone put “LLM” and “ESP32” in the same sentence. What’s new in 2026 is that the PLE-from-flash trick raises the ceiling on how much model you can purpose-train into an $8 part.

Try it: what running it actually looks like

The repo includes the firmware, the training pipeline, and the quantization scripts. The build follows the standard ESP-IDF flow (exact steps are in firmware/esp32_llm/README.md):

git clone https://github.com/slvDev/esp32-ai
cd esp32-ai/firmware/esp32_llm
# standard ESP-IDF build-and-flash cycle:
idf.py build
idf.py -p /dev/ttyUSB0 flash monitor

Expected result once it boots: the model streams a TinyStories-style story to the attached display (or serial monitor) at roughly 9.5 tokens per second — about reading pace. On the first run you’ll flash the 14.9MB quantized model into the 16MB flash alongside the firmware, which is exactly as tight as it sounds; boards with less than 16MB flash won’t fit it.

One practical warning from the spec sheet rather than the marketing: this needs an ESP32-S3 with 8MB PSRAM and 16MB flash (often sold as “N16R8”). Plenty of $5 boards ship with 2MB PSRAM and 4MB flash, and the model simply will not fit. Check the suffix before you order.

Why a home lab builder should care (and where this genuinely doesn’t matter)

You have a GPU server. Why would you care about a chip that generates children’s stories at 9.5 tok/s?

The power math is absurd, in a good way. An ESP32-S3 module draws on the order of 70mA in normal operation and ~120mA with Wi-Fi active per Espressif’s figures — roughly 0.23–0.4W at 3.3V. Run it 24/7 for a year and you’ve burned about 3.5 kWh: $0.42 at $0.12/kWh. Our home AI server power-bill breakdown put a typical GPU tower’s idle floor at 40–50W — 350+ kWh/year, roughly $42–$53 — just for existing. An edge node that handles the always-on 1% of work lets the expensive box sleep, or lets you skip waking it at all for trivial queries.

The two-tier pattern is the real architecture. The setup that makes sense isn’t “ESP32 instead of GPU,” it’s ESP32 in front of GPU:

  • Tier 1 (edge, always on): ESP32-S3 nodes at the sensor — a mic doing local wake-word/intent detection, a door-sensor node classifying “normal pattern or anomaly,” a camera trap flagging “worth a look.” Milliwatts each, nothing leaves the device.
  • Tier 2 (server, on demand): the flagged 1% of events goes over your LAN to the Ollama/llama.cpp box, where an 8B–70B model does the actual reasoning and synthesis.

Nothing in that pipeline touches the cloud, which is the entire point of this site. If you’ve read our local AI privacy audit, you know the weakest link in most “private” smart homes is the always-on cloud microphone; a $8 chip that classifies audio on-device and only ever forwards text over your LAN removes that link. For open-weight tiny models to purpose-train for tier 1, the FOSS side of this — datasets, licenses, training recipes — is well covered at aifoss.dev.

Where it doesn’t matter: if your home lab is one PC you turn on to run ComfyUI, none of this changes anything for you. Edge micro-models earn their keep when something must run continuously — and most home labs have exactly zero workloads like that until they wire up sensors, voice, or cameras. Don’t invent a use case to justify an $8 purchase; the drawer full of unused dev boards is a rite of passage, not a requirement.

Cloud contrast, for the record: the incumbent architecture for “AI in a cheap IoT device” is an API call — which means a subscription, latency, and a product that bricks when the vendor sunsets the endpoint. On-chip inference at $8 kills the recurring cost entirely. If your workload needs bursts of real GPU compute instead (training a custom micro-model, say), that’s a couple hours on a rented card — RunPod rents an RTX 4090 by the hour for less than the ESP32 board costs, and the trained 15MB artifact runs on-device forever after.

Honest verdict

The model is a demo; the memory architecture is the story. Judged as an LLM, a TinyStories generator is useless — and the author, to his credit, doesn’t pretend otherwise. Judged as an existence proof, it moved the ceiling on $8 hardware by two orders of magnitude in one release: 260K → 28.9M parameters on the same chip class, using a technique Google published for phones and someone ported down to flash-XIP on a microcontroller within a year.

For home lab builders the takeaways are concrete. First, “how many bytes per token, from which tier” is now the question that decides what runs on everything from your $8 sensor node to your 24GB GPU — internalize it once and half the 2026 model-fitting puzzles get easier. Second, purpose-trained micro-models on cheap edge silicon plus a real model on your server is a legitimately better architecture for always-on tasks than either extreme alone. Third: your RTX 3090 keeps its job. Nothing that fits in 16MB of flash is coming for it.

FAQ

Can I talk to this thing like a chatbot? No. It was trained on TinyStories and only writes short, simple stories. It doesn’t answer questions, follow instructions, or know facts. A 28.9M-parameter model has no room for any of that — the smallest models with real instruction-following are in the hundreds of millions of parameters, like LFM2.5-230M on a Pi 5.

What exactly do I need to buy to run it? An ESP32-S3 board with 8MB PSRAM and 16MB flash (the “N16R8” configuration is the common label). The bare boards run around $8 from Chinese sellers; US-shipped dev boards with the right memory config typically cost a few dollars more. A small SPI display is optional — output also goes to the serial monitor.

Is 9.5 tok/s usable? For streaming text to a screen, yes — it’s close to comfortable reading speed. For anything interactive or agentic, no. Compare 42 tok/s for LFM2.5-230M on a Raspberry Pi 5, or 80–100 tok/s for an 8B Q4 model on a used RTX 3090.

Why not just use a Raspberry Pi Zero for edge AI instead? A Pi Zero 2 W is 10× the price, boots an OS, draws more power, and wants an SD card that will eventually corrupt. For a fixed, single-purpose classifier the microcontroller wins on cost, power, and reliability. The Pi class earns its price the moment you need Linux, containers, or a model over ~50M parameters.

Does per-layer embeddings help my GPU server too? Same idea, different tier: Gemma 3n uses PLE to hold effective memory at 1.91B for a 5B model, and MoE expert-offload in llama.cpp applies the equivalent “only read what this token needs” logic between VRAM and system RAM. If you’re already offloading experts, you’re living the same principle.

Sources

Last updated July 27, 2026. Prices and specs change; verify current rates before purchasing.

Was this article helpful?