A Mathematical Framework for Transformer Circuits (2021)

Who Should Read

An LLM API or agent developer who wants an intuition for what happens inside transformers.

Core Mechanics

• They only dissected tiny models
  • They studied toy transformers with at most two layers and attention only, no MLPs.
  • GPT-3, by contrast, has 96 layers alternating attention and MLP blocks.
• Zero-layer models are bigram models
  • They predict the next token from the previous token alone.
  • The bigram table can be read straight from the weights without running the model.
• One-layer models ensemble bigrams and skip-trigrams
  • A skip-trigram is a pattern of the form "A… B C".
  • These tables can also be read from the weights without running the model.
  • Very simple in-context learning (learning from examples in the prompt) arises here.
• Induction heads appear at two layers
  • They are built by composing attention heads and implement a more general in-context learning algorithm.
  • The paper says these heads only developed in models with at least two attention layers.

Community Reactions

• Q/K/V is not the only way to see attention
  • A commenter said the equivalent larger matrices are more useful for interpretability than Q, K and V.
  • They argued that key/query analogies are not the only good way to understand attention.
• Redrawing diagrams around the residual stream
  • One commenter came to see it as a main bus from embedding to output, not just layer bypasses.
  • They now mentally redraw every architecture diagram with the residual stream at the center.
• Length is a barrier
  • One commenter tried many times but found it too long and asked if it is worth it.
  • Another predicted it will be seen as classic foundational work.

How to Apply

• Redraw an architecture diagram around the residual stream
  • Today, pick a transformer diagram and draw a straight line from embedding to unembedding.
  • Redraw attention and MLP as side blocks that add into that line.
• Test an induction pattern in a prompt
  • Paste the prompt below and check whether the model continues the earlier pattern.
  • The paper explains this behavior as in-context learning in small models.
• Split head roles in your model docs
  • In internal explanations, describe attention as where to look (QK) and what to move (OV).

Try It

# Paste into Claude or ChatGPT as is (simplified version of the paper's idea: following an induction pattern)

Follow the rule below without explanation.

moon -> 7
cable -> 3
river -> 9
cable -> 3
moon -> 7
river -> ?

Question: {{swap in a new pattern to test}}

Next to try: replace the examples with {{meaningless token pairs}} and see if the model still continues the pattern.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://transformer-circuits.pub/2021/framework/index.html