Show HN: TurboGPT: train 22KiB transformer in 13s

Who Should Read

A backend developer who only uses LLM APIs and wants to run a GPT training loop firsthand

Core Mechanics

• A byte-level tiny GPT trainer
  • It is an MIT-licensed CUDA C++ project.
  • The repo description says it trains a tiny GPT in under a minute (CUDA only).
• Checkpoints hold the full training state
  • runs/ctx4/ctx4.pt stores model, optimizer, scheduler, and trainer state.
  • Use --load CHECKPOINT.pt to resume.
• Logs are TensorBoard-compatible
  • One report is written per batch, capped at 8Mi reports.
  • report.json is derived from the log directory.
• 2.5295 BPB on hn1g
  • This is the result after 1.5G training tokens.
  • The source names no comparison model.
• Builds on Linux/NixOS and Windows
  • Linux uses nix-build; Windows needs VS2022 C++ tools and CUDA 13.4.

Community Reactions

• Opinion questioning the motivation
  • A commenter said they have seen a hundred similar projects.
  • They argued it has less learning value than Andrej Karpathy's.
• Opinion on model size naming
  • A commenter found it interesting that models are named by disk size rather than parameter count.
• Note about missing data
  • A commenter reported that hn1g.txt from the example is not available.

How to Apply

• Check your GPU's compute capability
  • Look up your GPU in NVIDIA's CUDA GPU list today.
  • On Windows, confirm that .\build.ps1 -CudaArch <value> builds.
• Run --data with your own text file
  • hn1g.txt is hard to get, so prepare a small text file.
  • Run with --log-to runs/test and inspect the TensorBoard logs.
• Test stop-and-resume with --load
  • Stop training midway and restart from the generated .pt checkpoint.
  • Check in the logs that optimizer and scheduler state carry over.

Try It

# Run in a terminal (CUDA environment): example based on the README's run instructions

# 1) Build (Windows, PowerShell, {{CUDA_ARCH}} is your GPU's value)
git clone https://github.com/lostmsu/TurboGPT
cd TurboGPT
.\build.ps1 -CudaArch {{CUDA_ARCH}}

# 2) Train (use your own text file)
.\build\turbogpt.exe --data {{your_text.txt}} --log-to runs/test

# 3) Resume training
.\build\turbogpt.exe --data {{your_text.txt}} --log-to runs/test --load runs/test/test.pt

# 4) Verify
python tests\verify.py

Terminology

Related Papers

• Show HN: Qwen3-4B with the AI writing tics trained out

  Distilling Qwen3-4B with LoRA cut its AI-writing slop score from 121 to 12.5 per 1000 words

• Dust: Pretraining Transformers Without Backpropagation

  Adding per-token noise to activations lets transformers pretrain without backpropagation, rivaling backprop.

• I had Gemini train its own replacement for $9

  Fine-tuning GLiNER on $9 of Gemini labels reached 0.83 F1 locally.

• Learning to solve hard problems in RL for LLMs by never giving up

  Cutting compute on easy problems and retrying hard ones in RL raised accuracy on the hardest problems

• Training a 3.8B LLM to 0.384 CORE for $998

  A 3.8B LLM trained on 8 B200s in 43 hours for $998 scored 0.384 on CORE.

• I were 17, I'd learn how to build LLMs from scratch

  PG says at 17 he'd skip startups and learn to build LLMs from scratch.

• Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it

Related Resources

• https://github.com/lostmsu/TurboGPT