Dust: Pretraining Transformers Without Backpropagation

TL;DR Highlight

Adding per-token noise to activations lets transformers pretrain without backpropagation, rivaling backprop.

Who Should Read

A backend developer using LLM APIs and agents who is curious whether LLM training can escape backprop.

Core Mechanics

Per-token noise estimates the gradient
- Dust adds independent Gaussian noise to each linear layer's output at every token and runs a forward pass.
- It rewards each noise by the per-token loss change and averages the reward-weighted noise to estimate the output error.
- The outer product of that estimate with the layer input gives the weight gradient.
Each token is a virtual population member
- Weight-space ES (evolution strategies: perturb weights and keep what works) evaluates one member per forward pass.
- Dust evaluates thousands of tokens at once, so one pass covers a population at least three orders of magnitude larger.
10^3 to 10^4 times more efficient than EGGROLL
- From 1M tokens up, this is versus a transformer implementation of EGGROLL, a state-of-the-art ES method.
- The figure comes from the authors' extrapolations.
Bigger models are more population-efficient
- A 243M-parameter model beats a 120x smaller model at most population sizes.
- This contradicts the belief that zeroth-order methods don't scale to large networks.

Community Reactions

Skeptics say gradients are simply more useful
- One commenter argues neural objectives are smooth, gradients help more as parameters grow, and a complexity gap between gradient-based and derivative-free methods has been proven for Lipschitz convex optimization.
- They point to Muon, an optimizer specialized to network structure, as a more promising direction.
Critique that the paper ignores convexity
- One commenter suspects Dust's advantages vanish without nonconvexity and that improved first-order methods could overtake it.
- They say zeroth-order is still a primary option when a network calls a simulator that exposes no gradients.
Experience: ZOO is useful where backprop is weak
- One commenter sees ZOO as a tool for regimes where backprop is fundamentally weak rather than a replacement.
- They cite a paper training large RNNs with CD-RGE without backprop-through-time and plan to try Dust on a personal project.
Easier to parallelize but costlier in compute
- One commenter asks if it is less compute-efficient than backprop but easier to parallelize.
- Another wonders whether fine-tuning a backprop-trained checkpoint with Dust could unlock further gains.

How to Apply

Read the TL;DR and Method section
- Spend 30 minutes on the Introduction and Section 2 today and note how Dust differs from backprop.
- Trace how equations (1)-(3) turn reward-weighted noise and the layer input into a gradient.
List training steps where backprop fails
- Find places where a non-differentiable simulator or external program sits in your training loop.
- Treat those as zeroth-order candidates, matching the use case commenters described.
Compare with backprop using the public code
- Clone the repo from the Code link at the top of the paper and run a small configuration.
- Compare validation loss against backprop at the same token budget and try populations 64, 1k, 16k.

Try It

copy & run

# Paste into Claude or ChatGPT as is (simplified version of the paper's method: concept check without running code)

You are an ML professor. Check whether I understood the Dust method described below.

[Dust summary]
- Add independent per-token noise σ·a_t to each linear layer output y_t and run a forward pass.
- Use the per-token loss change c_s as the reward, and average reward-weighted noise over K draws to estimate the output error.
- Sum the outer product of that estimated error with the layer input x_t over tokens to get the weight gradient.

[My understanding]
{{my understanding in 3-5 sentences}}

Compare with the summary, point out mistakes, and explain how it differs from backprop using a developer analogy.

Next to try: ask why the noise ratio of the gradient estimate shrinks as the population grows, at a beginner level

Terminology

BackpropComputes each weight's exact gradient by propagating the loss backward from output to input. It is like grading an exam and handing every step its precise share of blame.

Zeroth-order methodEstimates a direction by perturbing values and watching loss changes, without computing gradients directly. It is like finding downhill blindfolded by testing steps.

Node perturbationAdds noise to neuron outputs (activations) instead of weights. The argument is that the output space is smaller than the weight space.

ES (Evolution Strategies)Optimization that makes many random variants of the parameters, evaluates them, and moves toward the better ones. EGGROLL makes this cheaper with low-rank perturbations.

Virtual populationPopulation effect achieved by giving each token different activation noise instead of materializing weight copies. One forward pass evaluates thousands of candidates.