AI Chip Architectures

Who Should Read

A backend or AI service developer wondering why LLM inference cost and latency come down to chip design

Core Mechanics

• CPU gains fell from 52% to 3% a year
  • Single-threaded CPU performance grew 52% a year in the 1980s.
  • By 2018, with Moore's Law and Dennard Scaling ended, it was 3%.
  • That created the need for domain-specific architectures (DSAs).
• Google TPU v1 beat CPUs by 29x
  • It delivered 29x the throughput of a CPU on neural-network inference.
  • Its energy efficiency was 80x better than a CPU.
• The bottleneck is memory, not compute
  • AI compute is dominated by matrix multiplication.
  • Compute scaled exponentially but memory bandwidth did not (the memory wall).
  • Each chip takes a different strategy for moving data.
• Decode is far more memory-bound than prefill
  • Prefill is a large matrix-matrix multiply with high arithmetic intensity.
  • Decode emits one token at a time, becoming a matrix-vector product with much lower intensity.

Community Reactions

• A well-scoped primer, per commenters
  • One commenter skimmed architectures they knew and found no errors an LLM would typically make, calling it the best primer they had seen.
  • Another called it among the best deep-dives on the topic.
• Complaints that it looks AI-written and is hard to read
  • Several commenters said the text seems AI-generated.
  • Others complained about the thin font and overused italics.
• Interest in alternatives like analog and neuromorphic
  • Some said in-memory analog (EnCharge, Mythic) and RRAM/MRAM research is more exciting.
  • One asked whether analog or neuromorphic could make power-hungry hundred-kilowatt systems redundant.
• Laptop NPUs went unused, one commenter reported
  • A commenter said their brother's AI-chip laptop showed 0% NPU usage in Task Manager every time.

How to Apply

• Measure prefill and decode latency separately
  • Today, log time to first token and per-token generation time separately.
  • Compare long-prompt requests against long-output requests.
• Check long-context cost through the KV Cache
  • Raise concurrency and context length step by step and measure decode speed.
  • If speed drops, treat it as a bandwidth bottleneck and tune batch size.
• Turn on batching first when self-hosting
  • Confirm continuous batching is enabled in your serving engine.
  • If supported, enable speculative decoding and compare throughput.

Try It

# Paste into Claude or ChatGPT (prompt to apply the article to your service)

I run an LLM inference service. Based on the info below, diagnose whether prefill or decode is the bottleneck.

- Model/GPU: {{model and GPU}}
- Average input tokens: {{input length}}
- Average output tokens: {{output length}}
- Concurrent requests: {{concurrency}}
- Current metrics: time to first token {{TTFT}}, time per output token {{TPOT}}

Premise: prefill has high arithmetic intensity (compute-bound), while decode reads all weights and the KV Cache for every token, so it is memory-bandwidth-bound. Continuous batching and speculative decoding raise decode's arithmetic intensity.

Questions: 1) Where is the bottleneck? 2) Rank three settings to try. 3) Which metric confirms each one worked?

Next step: pick one suggested setting and compare real measurements before and after.

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Related Resources

• https://www.jepeake.com/ai-chip-architectures