LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques | AI Paper Digest

TL;DR Highlight

A public leaderboard cut LoRA fine-tuning from 11m57s to 22s on a fixed task and GPU.

Who Should Read

ML or backend engineers fine-tuning small models with LoRA who want to know which speedup tricks actually work.

Core Mechanics

Same task, same GPU, only time matters
- Fine-tune Qwen2.5-1.5B with LoRA to at least 57.0% on GSM8K.
- Hardware is one Modal L40S, and the shortest training wall-clock time wins.
Records are re-run three times
- Each record is re-run with fresh seeds, and all runs must clear the target.
- Official time is the mean of the three runs.
Track 1 went from 11m57s to 0m22s
- The baseline (#0) is LoRA r=16 on all linear layers, 3 epochs, cosine LR: 11m57s at 59.4%.
- The current record (#5) by @slippylolo takes 22s at 58.0%.
- It uses 12 optimizer updates: 8 through the full network, then 4 through only the top decoder layer.
The first win was packing plus loss masking
- It used sequence packing and completion-only loss masking.
- Result: 6m05s at 61.1% with 2 epochs, faster and more accurate than the baseline.

Community Reactions

Opinion that a single-task LoRA leaderboard may be too narrow
- A commenter doubted LoRA training time is enough of a bottleneck.
- They feared heavy overfitting with one task and one model.
- They suggested more tasks would help tricks transfer.
Opinion that resource limits drive creativity
- A commenter argued that limits force smarter solutions than just scaling weights and data.
- They compared it to urban growth boundaries.
Reactions to the name clashing with LoRa radio
- Several commenters first read the title as the wireless protocol.
- One also said the repo seems to lack an introduction paragraph.
Experience applying the speedrun approach to AI safety
- A commenter said they chose LoRA loss functions in a similar spirit.
- They reported distilling a Sparse AutoEncoder into a 5.3MB probe and shared a LessWrong writeup.

How to Apply

Run the baseline once on Modal
- Today, create a Modal account, clone the repo, prefetch, and run the baseline with `--runs 1`.
- Record your own training time and accuracy.
Add packing and completion-only masking to your pipeline
- If your fine-tuning data is mostly short samples, pack them to cut padding waste.
- Compute loss only on answer tokens, not the prompt.
- Compare time and accuracy before and after under identical conditions.
Try data pruning experiments
- Train on only the 4k shortest samples for under 1 epoch.
- Compare accuracy and time against full-data training.
Verify new tricks on a second task
- Check whether a trick that speeds up one dataset reproduces on another model and task.
- Do not make it a team default before that.

Try It

copy & run

# Run in a terminal: run the official baseline once on a Modal L40S

git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun
pip install modal pyyaml && modal setup

# Cache the model and data in a volume (one-time)
python harness/modal_verify.py --prefetch

# One baseline run (measures time and accuracy)
python harness/modal_verify.py --submission submissions/000-baseline --runs 1

# Next: copy submissions/TEMPLATE/, add packing or data pruning, and verify with --runs 3

Terminology

LoRALow-Rank Adaptation. A cheap fine-tuning method that freezes the base weights and trains a small adapter on top. Not the LoRa radio protocol.

wall-clockReal elapsed time measured by a stopwatch. Unlike FLOPs or step counts, it captures kernel and data-loading efficiency.

sequence packingConcatenating several short samples into one sequence to cut padding waste. Like packing a box with no gaps.

completion-only loss maskingExcluding the prompt from the loss so only answer tokens provide the learning signal.

modded-nanogptA public speedrun project to train a small GPT to a fixed target as fast as possible. This repo is its fine-tuning counterpart.

LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques