You could have invented transformers from memory

TL;DR Highlight

Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million

Who Should Read

An LLM API developer who lacks intuition for why transformers are shaped this way and what MLP and attention layers do

Core Mechanics

Every layer is a memory store
- A memory is one key-value pair: a key for lookup and a value to retrieve.
- A store that recalls by similarity is called an associative memory.
Linear layers store by outer products
- To add a memory (k, v), add v·kᵀ to the weights.
- Recall compares the query with every key and returns the similarity-weighted sum of values.
Interference drowns out memories
- A recall returns the target value (signal) plus values from other memories (interference).
- Capacity is the number of memories at which interference grows as large as the signal.
Linear capacity equals key dimension
- With random unit keys, the interference-to-signal ratio is about √(M/d_k).
- So capacity is M≈d_k, and beyond that perfect recall is impossible since only d_k orthogonal directions exist.
Capacity grows from linear to attention

Community Reactions

Math notation is a barrier
- One reader said they know no linear algebra, and another felt excluded because the notation is unexplained.
- Another reader said that after reading, they still could not have invented transformers.
Random keys give only a bound, some said
- The article assumes random keys, but real data has structure that allows more compression.
- So the conclusions are bounds rather than true estimates, and the real leverage comes from interaction with data structure.
Question about what the values are in an LLM
- A reader got lost at the final link from memory to LLMs.
- They asked: if context tokens are the keys, what are the values?
Author shared a summary thread
- The author posted a link to a short summary on X.
- Another comment liked that transformers find a suitable memory in one step, unlike multi-step vanilla Hopfield networks.

How to Apply

Simulate interference yourself
- In 30 minutes, build an outer-product store with M random unit-vector keys.
- Vary M around d_k=64 and check whether recall error matches √(M/d_k).
Check embedding dimension against index size in RAG
- If your index has far more chunks than embedding dimensions, check whether wrong documents leak into results.
- Use the interference idea as a hypothesis and compare against your retrieval logs.
Explain layers as memory stores in a team study session
- Draw linear, MLP and attention as one key-value lookup framework.
- Open the post's interactive app and code and move M live.

Try It

copy & run

# Run in Python: simple version (a simulation of the post's formulas, not the original code)
# pip install numpy
import numpy as np

d = 64
rng = np.random.default_rng(0)

for M in [8, 32, 64, 256, 1024]:
    K = rng.normal(size=(M, d))
    K /= np.linalg.norm(K, axis=1, keepdims=True)
    V = rng.normal(size=(M, d))
    V /= np.linalg.norm(V, axis=1, keepdims=True)
    W = V.T @ K
    out = W @ K[0]
    interference = np.linalg.norm(out - V[0])
    print(M, 'interference/signal =', round(interference, 2), 'vs sqrt(M/d) =', round((M / d) ** 0.5, 2))

# Next: make the keys mutually orthogonal (M<=d) and check that interference goes to about zero.

Terminology

Associative memoryA store that retrieves by similar content rather than address. Think of a hashmap where a near-match key still returns the closest item.

Outer productAn operation that turns two vectors into a matrix. Like stamping a key-value pair onto one shared sheet.

Interference-to-signal ratioHow much other memories contaminate the one you want. Like static getting louder than the radio broadcast.

OrthogonalDirections that do not affect each other at all. Orthogonal keys mean no other memory leaks into recall.

CapacityThe number of memories a store holds before interference grows as large as the signal.