I'm making LLMs work on small and old devices
TL;DR Highlight
Patched llama.cpp runs 16B MoE models on devices with about 1GB RAM
Who Should Read
A developer who needs small LLMs on edge or on-device hardware and wants to learn memory and speed tricks
Core Mechanics
- 16B model on 1GB RAM devices
- The Xiaomi TV Box S (3rd Gen) has about 1GB usable RAM and runs Ling-mini-2.0 16B at 5.1 tok/s.
- The iPad Air (2013) has about 700MB usable RAM and runs the same 16B model at 2.6-4.5 tok/s.
- Experts paged in from flash
- A mixture-of-experts (MoE) model uses only a few experts per token.
- Popular experts stay in RAM, the rest are read from flash on demand, and the next layer's are prefetched.
- Drop unused vocabulary rows to save RAM
- Most of a small model's vocabulary is rarely used, so those output rows are removed.
- On the TV, headless mode also stops the launcher and Google services.
- Kernels built for each chip
- They built a 1-bit lookup-table kernel and repacked 2-bit and ternary formats for the Cortex-A55.
- They also made versions for the A7, which lacks dot-product instructions.
- Everyday model differs from biggest model
- On the TV box the everyday model is Qwen3.5-2B at 7.7 tok/s.
- On the iPad Air it is Bonsai-1.7B (1-bit) at 24-26 tok/s.
- Everything is a llama.cpp patch
- The model files need the patched llama.cpp, and stock llama.cpp can't load them.
- The repo says every number was measured on the device itself.
Community Reactions
- Doubts about small-model usefulness
- Several commenters asked whether such small models are useful for practical tasks or just novelty.
- Opinion that Qwen is a poor tiny model
- One commenter said it is scaled down and falls apart on real work.
- They named MiniCPM, LFM, Granite and Gemma E2B as designed for tiny sizes.
- Experience with a 6.5M model doing many tasks
- One commenter trained a 6.5M-parameter model that handles reasoning, math, tool calling, RAG and chat.
- They said it is about 30MB in ONNX and runs with under 10ms latency in the browser.
- Opinion that agents can one-shot this work
- One commenter said inference engines are mostly multiplication, so modern agents can build one in a single prompt.
- They said the hard part is sourcing the hardware and giving ssh access.
How to Apply
- Pick candidate models for your device
- Check the usable RAM of your target device within 30 minutes today.
- Use that number to pick candidates such as Qwen3.5-2B or Bonsai-1.7B from docs/models.md.
- Try MoE models larger than your RAM
- Include MoE models among your on-device candidates.
- Check whether expert paging from flash lets you run a model larger than RAM.
- Compare small models on your own eval set
- Build a per-task eval set that includes tool calls and loop failures.
- Measure quality on the device alongside speed (tok/s).
Try It
# Paste into Claude Code or ChatGPT (a simplified version based on the prompt idea from the comments)
Target device: {{device model, CPU, usable RAM}}
Model: {{e.g. qwen3 0.6b}}
Build an inference engine that runs on this device.
- Write a script that attaches the weights to the binary
- Take a stream of u32 token IDs on stdin and stream u32 token IDs on stdout
- Tell me how to test it first in a simulator such as qemu
Next step: measure speed (tok/s) and ask it to explain where the bottleneck isTerminology
Related Papers
Gemini 4 Neon
Gemini 4 Neon cuts cost per task 32% versus Carbon, at $3 input and $15 output per million tokens
AI is killing All About Berlin
After Google AI Overviews, a Berlin guide site lost 70-75% of its traffic.
Not Everything Needs an LLM
PostHog used small decision models to skip 98.7% of expensive LLM calls
GPT‑6.2 Astra and Sol
GPT-6.2 Astra and Sol cut costs sharply, but over-engineering and early-stop reports are mixed
Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
One trn2.48xlarge serves GLM-5.3-Flash through an OpenAI-compatible API using NKI megakernels.
Know When to Zoom: How Small Is Too Small for Vision-Language Models?
Splitting large photos into tiles raised Claude and Gemini accuracy by 11.1–14.5 points
Understanding Positional Embeddings in Modern LLMs
A walkthrough of how LLMs encode token position, from GPT-2 to RoPE
Show HN: DeepSeek V3 in ~200 lines of pure Jax
DeepSeek V3's core design in 190 lines of JAX, trained on a MacBook Air in 2.6 hours