Running a 28.9M parameter LLM on an $8 microcontroller
TL;DR Highlight
Keeping most parameters in flash lets an ESP32-S3 generate text at 9.88 tok/s
Who Should Read
A developer looking for ways to run a local LLM on edge or embedded hardware
Core Mechanics
- Most of the model stays in flash
- 25M of the 28.9M parameters sit in a lookup table in flash.
- Each token reads only about 6 rows, around 450 bytes.
- Per-Layer Embeddings are the key
- The idea comes from Gemma 3n and is adapted to a microcontroller's memory layout.
- Most of the model is never loaded into RAM and is sampled a little at a time.
- Memory is split by access frequency
- SRAM (512KB) holds activations and norm weights read many times per token.
- PSRAM holds the core and output head, and flash holds the 25M table.
- Speed is 9.88 tok/s
- This is end to end on an ESP32-S3 (512KB SRAM, 8MB PSRAM, 16MB flash).
- Compute is 94.9 ms/token and the 4-bit model is 14.9MB.
- Capability is very limited
- Trained on TinyStories, it writes short, simple stories that mostly stay coherent.
Community Reactions
- A question about scaling via flash came up
- One commenter asked whether larger models could run on flash-backed CPUs with good access patterns.
- Flash wear was raised as a concern
- A commenter asked whether constant reads would wear out the flash.
- TTS use was suggested
- One commenter noted 20-30M parameter TTS models exist, so an offline ESP32 could read aloud in near real time.
- Training got praise
- One commenter was more impressed by how the weights were trained than by the on-device run.
How to Apply
- Flash the TinyStories model to a board
- With an ESP32-S3 board, run scripts/fetch_model.sh tinystories to download the model.
- Then run scripts/deploy.sh tinystories to flash it.
- Draw a memory-tier placement table
- List your model's tensors by access frequency.
- Mark large tables read only once or twice per token as candidates for slow storage.
- Read the RESULTS.md ablations to scope your use
- Check the measured numbers before designing an on-device model.
- Decide whether simple generation is all you need.
Try It
# Paste into Claude/ChatGPT (simplified version of the paper's method: memory tier check)
I want to run {{model name, parameter count}} on {{target device}}.
Device memory: SRAM {{size}}, PSRAM {{size}}, flash {{size}}.
Using the rules below, make a table of where each tensor should live.
- Values read many times per token (activations, norm weights) go in the fastest memory
- The core and output head, read once per position, go in mid-tier memory
- Huge embedding tables where only a few rows are read per token go in the slowest, largest memory
Also list each item's estimated size and bytes read per token.
Next step: find the item with the most bytes read per token and ask how to reduce itTerminology
Related Papers
Mercury 3: Diffusion at scale
Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality
Timestral: Mistral's time-series foundation models
A 209M time-series model ranks first on three benchmarks and runs on CPU.
DeepSeek V4.1 Flash 1017 (Max)
DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.
If you don't run your own LLM proxy, how do you know whats going on?
Running your own LLM proxy shows which keys, models, and cache misses drive your usage.
DeepSeek: DeepSeek v4.1 Flash 1017
DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents
DeepSeek-v4.1-Flash-1017
DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length
Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware