Learning to solve hard problems in RL for LLMs by never giving up

Who Should Read

An ML engineer doing RL post-training of LLMs with GRPO who only watches average eval scores and can't get hard problems solved

Core Mechanics

• Average scores hid the hard problems
  • In Olmo 3.1 RL-Zero Math's AIME 2025 eval, problems with initial pass@32=0 mostly stayed at pass@32=0.
  • Most gains came from easy problems going from somewhat solved to mostly solved.
• Matthew Effect: RL helps what's already easy
  • The author names the pattern where RL gains scale with the model's initial competence the Matthew Effect (the rich get richer).
  • It also showed up in code and agentic RL, using logs from Deepcoder and DeepSWE.
• Bigger k doesn't fix it
  • Training Qwen 2.5 0.5B Instruct with GRPO on GSM8k platinum, k=4 with 64 prompts solved harder problems than k=32 with 8 prompts.
  • k is completions sampled per prompt; the author blames wasted compute on easy problems (signal efficiency) rather than just signal loss when all completions are wrong.
• k=32 wastes compute on rare wrong answers to easy problems
  • Before about step 200, k=32 was better at finding rare correct solutions to hard problems.
  • After that k=4 pulled ahead, since k=4 filters a problem at 4/4 correct while k=32 needs 32/32.

Community Reactions

• Experience of Claude confirming a wrong performance number
  • One commenter said Claude confirmed their WebSocket benchmark of 3,250,000 messages/sec, but the number turned out wrong two weeks later.
  • After correcting it, the rate was 100,000 messages/sec, and queue changes brought it to 800,000.
• Note that the top links don't lead to the paper or code
  • A commenter pointed out that the arxiv and github links at the top both go to the same page.

How to Apply

• Split your eval set by initial pass@k difficulty
  • Sample the initial model many times and separate pass@32=0 problems as hard from the rest.
  • Plot per-bucket accuracy before and after training to see whether the average hides the hard problems.
• Compare smaller k with more prompts
  • Fix the batch size and sweep k over 4, 8, 16, 32, comparing accuracy on the hard bucket.
  • Also log what share of each batch is easy prompts.
• Add an NGU retry loop to GRPO sampling
  • Sample a small k, and for prompts with all wrong answers put them back with probability p for k more samples.
  • Train on all accumulated completions once solved, and filter old completions by an age threshold (T=4 worked best in the paper).
• Keep stale completions in the GRPO baseline
  • Use filtered stale completions for the baseline even if you don't train on them, and rescale the negative advantages so total reward is 0.
  • The paper found this beats ignoring them or leaving the baseline non-zero.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's idea: split eval results by difficulty)

I have LLM eval results. The average looks improved, but I want to check whether hard problems actually got better.

[Initial model results]
{{per-problem pass@k for the initial model}}

[Post-training results]
{{per-problem pass@k after training}}

Please do the following:
1. Label problems with initial pass@32=0 as hard, and split the rest evenly into medium and easy by pass rate.
2. Make a table of accuracy change per bucket before and after training.
3. Explain which bucket the average gain came from and whether hard problems really improved.

Next step: if gains are concentrated in easy buckets, suggest comparing a setup with smaller k and more prompts.

Terminology

Related Papers

• Show HN: Qwen3-4B with the AI writing tics trained out

  Distilling Qwen3-4B with LoRA cut its AI-writing slop score from 121 to 12.5 per 1000 words

• Dust: Pretraining Transformers Without Backpropagation

  Adding per-token noise to activations lets transformers pretrain without backpropagation, rivaling backprop.

• Show HN: TurboGPT: train 22KiB transformer in 13s

  A CUDA C++ project trains a tiny byte-level GPT in under a minute.

• I had Gemini train its own replacement for $9

  Fine-tuning GLiNER on $9 of Gemini labels reached 0.83 F1 locally.

• Training a 3.8B LLM to 0.384 CORE for $998

  A 3.8B LLM trained on 8 B200s in 43 hours for $998 scored 0.384 on CORE.

• I were 17, I'd learn how to build LLMs from scratch

  PG says at 17 he'd skip startups and learn to build LLMs from scratch.

• Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it

Related Resources

• https://mnoukhov.github.io/posts/ngu/
• https://arxiv.org/abs/2609.13443