OpenTPU – An open-source AI accelerator, developed by AI | AI Paper Digest

TL;DR Highlight

An AI-designed open-source accelerator runs real LLMs like Qwen3 on an FPGA card

Who Should Read

A backend or AI-service developer who wants to see how LLM inference works on hardware from the ground up

Core Mechanics

Models really run on the AI-made chip
- Ten models with real weights run on a Kintex-7 PCIe card.
- The card's tokens match the simulator bit for bit.
DRAM bandwidth bounds decoding
- LFM2.5-230M int8 decodes at 59.0 tok/s using 14.5 GB/s of DRAM, 85% of peak.
- Decode is DRAM-bound, and LiteDRAM reads at 82-85% of the DDR3 peak.
4-bit weights speed decode by 40%+
- They use FP4 values with two-level block scales, 4.25 bits per weight.
- Bytes per token drop by about a third, and decode is 40% (Qwen3.5) to 45% (Qwen3, LFM2) faster.
- Perplexity (how confused the model is about the next token) gets measurably worse.
MoE models bigger than the card run too
- Qwen3.5-35B-A3B (34.7B parameters) reaches 3.95 tok/s.
- The host copies only the needed experts over PCIe, streaming 153 MB per token.

Community Reactions

A question about why labs don't burn frontier models into chips
- A software developer asked, saying they don't understand the economics or physical limits.
A pushback that "AI developed its own inference hardware" is overstated
- The commenter said a human had an LLM build a simulation environment for hardware design.
- Then the LLM optimized designs against constraints when prompted by a human.
A comment asking whether performance beats state-of-the-art hardware
- It noted there is no comparison with the best human-designed hardware.
- Another asked for details on the apparently ~$300 FPGA board.
An opinion that an experienced user steering the LLM matters
- The commenter said a lot is gained from pointing an LLM in a tasteful direction.
An experience reading the RTL and learning that LLMs apparently don't need correct floating-point operations

How to Apply

Run otpu-chat on the simulator
- Today, run pip install -e ., download LFM2.5-230M, and run otpu-chat --model lfm2 --backend isa.
- You can feel out the accelerator's structure on a laptop without a card.
Compute bytes per token for your model first
- Decode speed is bound by DRAM, so divide memory bandwidth by model size to get an upper limit.
- Compare that limit with measured speed before applying 4-bit quantization.
Add golden-comparison checks before quantizing
- Write a script like tools/validate.py that measures top-1 agreement and KL against fp32 output.
- Log these numbers whenever you change quantization settings.
Learn to read a Lens trace
- Follow docs/lens.md and open a Qwen3 decode trace.
- Use the roofline to separate compute waits from DRAM waits and find the bottleneck.

Try It

copy & run

# Paste into a terminal (runs on the simulator without a card; Verilator is only needed for RTL tests)

pip install -e .
pip install pytest torch transformers

hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M
otpu-chat --model lfm2 --backend isa

# Next: compare the same model's simulator output with the CPU golden
python3 tools/validate.py --model lfm2 --wformat fp4 --head-format int8

Terminology

systolic matrix unitA circuit where data flows through a grid of compute cells like a wave to do matrix multiplication. Think of a conveyor belt where each worker does one step.

RTLDesign code (here SystemVerilog) that describes a chip's circuits clock by clock. It is essentially the circuit's source code.

FPGAA chip whose wiring can be reconfigured. Like Lego, you can build a circuit and rebuild it.

MoEA model structure with many experts where only some are used per token. Like a big hospital calling only the relevant specialists for each patient.

rooflineA chart showing whether compute limits or memory bandwidth limits is the bottleneck.

ISAThe set of instructions a chip understands. It is the contract between software and hardware.

OpenTPU – An open-source AI accelerator, developed by AI