Training a 3.8B LLM to 0.384 CORE for $998

Who Should Read

A developer training a small LLM on their own who wants to know where to spend budget on optimizer, precision, and context length.

Core Mechanics

• A 3.8B model for $998
  • It trained on 65.3B tokens in 43 hours on 8× B200 and scored 0.384 on CORE.
  • In the post's table, nanochat d32 costs about $1000 and takes about 33 hours for CORE 0.310.
• The first 858M run failed
  • It trained about 5.8 days on one A100 and got 60.45% PIQA, below GPT-2 124M at about 63%.
  • The author blamed an overly aggressive cosine decay, a conservative learning rate, and AdamW everywhere.
• Five changes improved it a lot
  • They were a trapezoidal LR schedule, Muon for matrix params, ClimbMix data, FP8, and 1024 context.
  • Muon is about 25% slower per step but only about 4% at 7 gradient-accumulation steps.
• Throughput tuning cut cost
  • FP8 plus vocab padding gave +33% throughput.
  • bf16 master weights cut VRAM by 27% and raised throughput from 640K to 1.4M tok/s on the 1.5B config, with CORE 0.22 vs 0.23 at 4,000 steps.

Community Reactions

• Opinion on trying newer ideas
  • One commenter wanted to redo this at about 1B params with gated delta nets, n-gram embeddings, and gated residuals.
• Reproducibility concern
  • One commenter could not find a link to the model code or training data.
• Idea for using small models in products
  • One commenter suggested using Claude for edge cases and sample data, then training a tiny LLM on demand to call Claude less.
• Request for a follow-up
  • One commenter asked how much it would cost to distill from Qwen's best roughly 30B model.

How to Apply

• Check whether eval prompts get truncated
  • Today, measure token lengths of your few-shot eval set and see how many exceed max_seq_len.
  • Look at heavily truncated tasks separately when comparing scores.
• Switch small-model training to a trapezoidal LR
  • Replace cosine with 5% warmup, a flat hold, and a linear cooldown to 5% of peak over the last 50%.
  • Check that eval loss is still falling at the final step.
• Use local shards instead of streaming
  • Before any multi-hour paid GPU run, download the data shards first.
  • The post saw 2-3% more throughput and avoided network hangs.
• Ablate with one-line config diffs
  • Make the optimizer, dataset, and value embeddings config values.
  • Compare loss and CORE against your baseline run at the same step.

Try It

# Paste into Claude or ChatGPT (simplified version of the post's method: training config review)

I am training a small decoder-only LLM myself. Review the training config below.

{{training config YAML}}

Check each item and suggest changes in a table.
1. LR schedule: if it is cosine, what the config looks like with 5% warmup, a flat hold, then linear decay to 5% of peak over the last 50%
2. Optimizer: whether matrix params can use Muon and the rest AdamW
3. Precision: whether FP8 is usable and whether the vocab needs padding to a multiple of 64
4. Eval: whether any few-shot tasks have prompts longer than max_seq_len
5. Data: the benefit of switching from streaming to local shards

Next step: ask it to write an ablation plan as a YAML diff of 3 lines or fewer, comparing eval loss at the same step.

Terminology

Related Papers

• Show HN: Qwen3-4B with the AI writing tics trained out

  Distilling Qwen3-4B with LoRA cut its AI-writing slop score from 121 to 12.5 per 1000 words

• Dust: Pretraining Transformers Without Backpropagation

  Adding per-token noise to activations lets transformers pretrain without backpropagation, rivaling backprop.

• Show HN: TurboGPT: train 22KiB transformer in 13s

  A CUDA C++ project trains a tiny byte-level GPT in under a minute.

• I had Gemini train its own replacement for $9

  Fine-tuning GLiNER on $9 of Gemini labels reached 0.83 F1 locally.

• Learning to solve hard problems in RL for LLMs by never giving up

  Cutting compute on easy problems and retrying hard ones in RL raised accuracy on the hardest problems

• I were 17, I'd learn how to build LLMs from scratch

  PG says at 17 he'd skip startups and learn to build LLMs from scratch.

• Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it

Related Resources

• https://hugovergnes.github.io/little-lm-3-8b/