Quadrillion-parameter transformers and how to train them
TL;DR Highlight
On fixed data, much bigger models with scaled weight decay and ensembling keep loss falling as compute grows.
Who Should Read
A developer using LLM APIs and agents who wants to understand how pretraining scaling laws change when data runs short.
Core Mechanics
- Fixed data means bigger models
- With a fixed dataset, the compute-optimal model size keeps growing with compute budget.
- Chinchilla fixes model size at 20 tokens per parameter in the same setting.
- About 7,000x Chinchilla at 5T tokens
- Extrapolated to 5T tokens, the law prescribes ~7,000x Chinchilla's parameter count at 1029 FLOPs.
- The ensemble holds over 300 parameters per token.
- Bigger models need much more weight decay
- Optimal weight decay grows steeply with epochs and model size.
- MuP covers learning rate but not weight decay.
- Hyperparameter law holds at 16x compute
- It stays near-optimal at 16x the fitted compute, within 0.01 nats of optimal loss on every axis tested.
- Each prior heuristic misses on at least one axis.
- Ensembling becomes a scaling axis
- Snapshot ensembling averages checkpoints from one run, so it costs no extra training compute.
- Deep ensembling (independent seeds) becomes optimal when compute is large relative to data, with member count growing roughly linearly in compute.
- Prior recipes get worse with compute
- On 100M tokens, the prior recipe's loss rises as compute grows.
- Hyperparameter transfer plus ensembling keeps loss falling.
Community Reactions
How to Apply
- Recheck how you size models
- If you fine-tune or train on fixed data, don't rely on the 20:1 Chinchilla ratio alone; vary model size and epochs together.
- Adjust weight decay along with model size and epochs.
- Try snapshot ensembling on multi-epoch runs
- Use a cyclic learning rate with one-epoch cycles late in training and save a checkpoint at each cycle's end.
- Average the checkpoints' predictions and compare validation loss with the single model.
- Spend extra compute on deep ensembles
- After growing model size and epochs, train additional independent models with different seeds and average their predictions.
Terminology
Related Papers
Astra can read and write weights to improve Qwen
An LLM edited Qwen3.5-0.8B's weights directly, lifting MATH-500 from 7.9% to 48.1%.
Show HN: Qwen3-4B with the AI writing tics trained out
Distilling Qwen3-4B with LoRA cut its AI-writing slop score from 121 to 12.5 per 1000 words
Dust: Pretraining Transformers Without Backpropagation
Adding per-token noise to activations lets transformers pretrain without backpropagation, rivaling backprop.
Show HN: TurboGPT: train 22KiB transformer in 13s
A CUDA C++ project trains a tiny byte-level GPT in under a minute.
I had Gemini train its own replacement for $9
Fine-tuning GLiNER on $9 of Gemini labels reached 0.83 F1 locally.
Learning to solve hard problems in RL for LLMs by never giving up
Cutting compute on easy problems and retrying hard ones in RL raised accuracy on the hardest problems
Training a 3.8B LLM to 0.384 CORE for $998
A 3.8B LLM trained on 8 B200s in 43 hours for $998 scored 0.384 on CORE.
I were 17, I'd learn how to build LLMs from scratch
PG says at 17 he'd skip startups and learn to build LLMs from scratch.