ESP32S3 cluster running 1.58-bit (BitNet) Language model
TL;DR Highlight
A 7-node ESP32-S3 cluster runs a 0.4B BitNet LLM over an SPI daisy-chain
Who Should Read
Embedded or backend developers wondering whether an LLM can run on microcontrollers and edge devices
Core Mechanics
- The model is split across 7 ESP32-S3 boards
- One board is the master and the other six are compute nodes.
- The repo description says 0.4B while the README body says 0.5B.
- Master handles I/O, nodes handle transformer layers
- The master runs the BPE tokenizer, token embedding, final RMSNorm, LM Head and greedy sampling.
- Nodes 1-6 each run 4 transformer blocks (layers 0-23), covering attention and MLP.
- Boards are linked by an SPI daisy-chain
- The FP32 hidden state goes from the master through nodes 1-6 and back to the master as a pipeline.
- Each node keeps its KV cache in PSRAM.
- It uses 1.58-bit ternary quantization (weights are -1/0/+1)
- Attention Q/K/V/O and MLP Gate/Up/Down projections are 1.58-bit.
- Multiply-accumulate is hand-optimized in assembly, and lookup tables are used.
- The model was shrunk to fit the ESP32
Community Reactions
- A view that it is closer to a toy
- One commenter felt the compression level makes it a fancy noise-maker, though still charming.
- Questions about practical use
- Commenters asked whether it could handle grammar checking in a basic word processor or generate worlds for text games.
- They said they did not know what such a cluster can do.
- Someone building something similar
- One commenter said they are working on a similar project with a 150M parameter model and less quantization.
- Question about cheaper hardware
- One commenter asked whether a Mac Mini is the lowest-cost option, or whether mini-ITX iGPUs or Pi HAT AI chips could run LLMs.
How to Apply
- Clone the repo and read workflow.md first
- Today, clone it and read the wiring, flashing and model-prep steps in workflow.md.
- Check whether you need 7 boards and estimate the hardware cost.
- Walk through the quantization pipeline in python_tools
- Read crop_token.py, crop_model_weight.py, qat_158.py and bit4_embedding.py in that order.
- Check whether vocabulary pruning and INT4 embeddings could apply to your own model.
- Use the layer-split pipeline as an edge design reference
- When distributing a model over small-memory devices, split layers evenly and pass the hidden state in sequence.
Try It
# Paste into Claude/ChatGPT (simplified version of the repo's method: checking an edge LLM split design)
I want to distribute a small LLM across several microcontrollers, using this structure as a reference.
- 1 master: tokenizer, token embedding (INT4), final RMSNorm, LM Head, greedy sampling
- 6 nodes: 4 transformer blocks each (24 layers total), 1.58-bit weights, KV cache in external RAM
- Link: SPI daisy-chain, FP32 hidden state passed between layers
My setup: {{board/chip type and count}}, model: {{model name and parameter count}}
Tell me how to split the layers for my setup and list the likely memory and communication bottlenecks as a checklist.
Next step: ask it to calculate how much data the hidden state transfers per generated token.Terminology
Related Papers
Mercury 3: Diffusion at scale
Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality
Timestral: Mistral's time-series foundation models
A 209M time-series model ranks first on three benchmarks and runs on CPU.
DeepSeek V4.1 Flash 1017 (Max)
DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.
If you don't run your own LLM proxy, how do you know whats going on?
Running your own LLM proxy shows which keys, models, and cache misses drive your usage.
DeepSeek: DeepSeek v4.1 Flash 1017
DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents
DeepSeek-v4.1-Flash-1017
DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length
Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware