I trained a small transformer in 1.5hrs and it beats many LLMs

Who Should Read

A backend or ML-adjacent developer who mostly uses LLM APIs and wants to weigh training a small model from scratch against them on cost and accuracy

Core Mechanics

• It's a small transformer, not an LLM
  • It was trained from scratch in 1.5 hours on a single 5090.
  • It scored 44% on the ARC-AGI-1 public eval at a cost of 67 cents.
  • The author says it matches TRM/HRM.
• Each input-output pair becomes a token sequence
  • Each puzzle gets its own learned additive embedding.
  • Because each sequence has two 2D grids, positions use learned 3D RoPE.
  • Color and dihedral augmentations are applied, then inverted at inference, and the 2 most common outputs are submitted.
• Representation mattered most
  • Switching 3D RoPE to 1D dropped the score to about 24%.
  • Removing the per-task embedding also dropped it to about 24%.
• Dropping input-token loss raised 40% to 44%
  • The loss now covers output tokens only, which makes it supervised.
  • The author says they don't understand why it helps.

Community Reactions

• Agreement that LLM gains come from post-training
  • One commenter said this applies to every benchmark.
  • Another said even private tests require trusting providers not to log and train on them.
• Pushback that 67 cents is misleading
  • One commenter said spending $100 would hit a ceiling fast, not give much better results.
  • They also asked what comes next after a custom model does decently on one task set.
• Debate over training on eval inputs
  • One commenter called it teaching to the test, a grey area.
  • Another said that if questions were seen one at a time, 44% wouldn't be reached.
• Eval puzzles share a generating process
  • One commenter said the model uncovers that shared process.
  • They suspected unrelated or garbage questions would reduce performance.
• Author clarifications and title complaints

How to Apply

• Reframe one of your tasks as input-output pairs
  • Pick a repeated classification or transformation task and collect 20 example pairs within 30 minutes.
  • That list is the starting point for judging whether a small model can learn it.
• Clone mdlARC and measure the run cost
  • Download the public code and run it on one GPU, logging time and cost.
  • The article puts about 2 hours on a 5090 at 67 cents.
• Apply loss to output tokens only and compare
  • If you have a sequence model, run it twice with and without input loss under identical settings.
  • Record both test loss and actual score.
• Match compute scope when comparing benchmarks
  • Note in your table whether pretraining cost is included.
  • Comparing total lifetime compute, as this article does, is the fair way.

Try It

# Paste into Claude or ChatGPT (simplified version of the article's method: checking whether a small model can learn input-to-output pairs)

I run a service that handles {{task description}}. Input-to-output examples: {{20 examples}}.

Using this article's approach, tell me:
1. How to turn these examples into token sequences
2. Augmentations that multiply the data without changing meaning
3. An experiment comparing loss on output tokens only vs. input plus output
4. Criteria for choosing between training a small transformer from scratch and using an LLM API

Next step: ask for a training-script skeleton based on that experiment design.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://mvakde.github.io/blog/44-on-arc-1/
• https://github.com/mvakde/mdlARC