Autoregressive Language Model on the 6502 Processor

Who Should Read

A developer curious about quantization and on-device inference who wants to fit an LLM into extremely limited memory and compute

Core Mechanics

• The model was designed around the 6502
  • It runs on a 1975 8-bit CPU with 32KB RAM.
  • Inference code (9KB) and weights (13KB) fit in 25KB of user space.
  • The CPU has no multiply instruction.
• BitNet turns multiplies into adds and subtracts
  • Weights are quantized to {-1, 0, 1}.
  • An 8x8 multiply-accumulate costs 150 cycles, a ternary accumulate 30.
  • Each parameter takes 1.58 bits.
• Four parameters are packed per byte
  • Five per byte is denser but needs divide-by-3 to unpack, which is costly on the 6502.
  • Four per byte uses 2 bits each and unpacks with a right shift.
  • 13KB holds about 52k parameters.
• Mamba was chosen over attention
  • Attention's KV cache grows with context and would not fit in 32KB RAM.
  • Mamba keeps a fixed-size state vector.

Community Reactions

• Opinion that banked memory should be allowed
  • Streaming weights in pages, as was period-appropriate, could support a bigger model.
• Opinion that assembly beats cc65
  • A commenter with NES experience says the 6502 suits C poorly and hand-written assembly could save space and time.
• Experience that Markov chains beat tiny neural models
  • A commenter says modern ML models scale down poorly and a simple Markov chain variant is almost always better.

How to Apply

• Try ternary quantization on your own model
  • Today, wrap one Linear layer's weights with the ternary_quantize below and run a forward pass.
  • Compare output quality against the original.
• Pick fixed-state architectures for edge deployment
  • If memory is tight, add recurrent models like Mamba to candidates instead of attention.
  • First check that memory stays flat as context grows.
• Replace softmax with a lookup table
  • Where exp is unavailable, precompute exp of the logit-max difference.
  • Build one table per temperature and sample with integers.

Try It

# Run in a PyTorch environment (pip install torch)

import torch

def ternary_quantize(w: torch.Tensor) -> torch.Tensor:
    q = torch.clamp(torch.round(w), -1.0, 1.0)  # forward: {-1,0,1}
    return w + (q - w).detach()  # backward: full precision gradient

w = torch.randn(4, 4, requires_grad=True)
x = torch.randn(4)
y = ternary_quantize(w) @ x
y.sum().backward()
print(ternary_quantize(w))
print(w.grad)

# Next: apply it to a small MLP's Linear weights, train it, and compare accuracy

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Related Resources

• https://mattbeton.com/blog/bitnet-6502.html