Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)

TL;DR Highlight

Keeping all weights on-chip gave a $250 FPGA about 21,300 tok/s.

Who Should Read

A backend developer running LLM APIs or serving who wants to understand why inference hits memory bandwidth limits.

Core Mechanics

Reading, not compute, is the bottleneck
- Each token requires reading every model weight once.
- On the KV260, the Arm cores and the fabric share one ~20 GB/s DDR controller.
- If weights live in DDR, the FPGA fabric gains nothing.
Put the whole model on-chip
- A 3.16M-parameter INT4 transformer (~1.5 MB) sits in on-chip URAM/BRAM.
- No DRAM is touched in the token loop.
- The author says this is the same idea behind Taalas, Groq and Cerebras.
Same model: 11 tok/s on the board's CPU
- The same model gets 11 tok/s on the board's Arm cores and 719 tok/s on a laptop RTX 3050 Ti.
- The fabric record is 59,965 tok/s, bit-exact.
The record build can't chat
- 16 streams share one weight pass for 59,965.5 tok/s at 200 MHz.
- The attention window is one token, so output is "he he he he he".

Community Reactions

Opinion that such projects get less HN attention
- A commenter notes that posts about running 1T+ models at seconds per token get optimism.
- They call this project conceptually brilliant as a demo despite poor scalability.
Opinion that FPGA barriers are high and GPUs scale better
- A commenter cites timing diagrams, sequential/combinational logic and long EDA runs.
- They say Yosys is still years behind Xilinx tools.
Opinion that ROM weights in ASICs block model updates
- A commenter says SRAM is not dense enough and DRAM has bandwidth issues.
- They float ideas like systolic arrays with fixed routing and programmable weights.
Question about flat throughput at 2,000 connections
- A commenter asks what per-user latency looks like when all share one stream.
Skeptical take on a model this small
- A commenter says the output is about as coherent as random.choice and slower.

How to Apply

Check whether your serving is bandwidth-bound
- Divide your model's weight size (GB) by memory bandwidth (GB/s) to get an upper bound on tokens/s.
- Compare it with your measured tok/s.
Compare INT4 vs higher precision
- Run INT4 and a higher-precision build of the same model on the same prompts and compare tok/s and quality.
- Check whether smaller weights read less and run faster.
Set a model size limit for on-device use
- Look up the on-chip or cache memory of your target hardware.
- Pick a model size so the model and KV cache fit.

Try It

copy & run

# Paste into Claude or ChatGPT (simple calculation prompt based on the article's idea)

I want to check whether my LLM inference speed is bound by memory bandwidth.
- Parameter count: {{parameters}}
- Weight precision (bits): {{FP16/INT8/INT4}}
- Memory bandwidth (GB/s): {{bandwidth}}
- Measured tok/s: {{measured}}

Assuming all weights are read once per token, compute the bandwidth-based theoretical max tok/s, compare it to my measurement, and explain where the bottleneck is.

Next step: ask how the theoretical max changes if I switch to INT4.

Terminology

FPGAA chip whose wiring you can reconfigure to build custom digital circuits. Unlike a CPU running software, you rewire the hardware.

On-chip memory (URAM/BRAM)Very fast memory inside the chip. Like keeping books open on your desk instead of walking to the warehouse (DDR).

INT4Storing weights as 4-bit integers (16 levels). It coarsely rounds numbers to shrink size.

KV cacheMemory holding previously computed attention values. A notepad that avoids recomputing them.

Memory bandwidthHow fast data can be read from memory. Think of the width of a water pipe.