What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training

Who Should Read

ML engineers who post-train models with RL or fine-tuning and judge improvement by pass@k alone

Core Mechanics

• pass@k only sees correct probability
  • With the same per-problem correct probability p, pass@k is identical at every k.
  • How correct mass is spread across distinct solutions is not in the formula.
• Same model, opposite diversity shifts
  • Qwen2.5-1.5B-Instruct was trained with GRPO and with RFT (training on its own shortest verified-correct output).
  • RFT raised token entropy, answer entropy, and unique answers, while GRPO lowered all three.
  • There was no overlap across all three seeds.
• Correct-only solutions were less diverse under GRPO
  • Distinct-3 among correct solutions was 0.436 for GRPO versus 0.512/0.509 for RFT.
  • Truncating to equal length left a similar 14% gap.
• Only pass@1 separated the arms
  • On GSM8K pass@8 and pass@32, confidence intervals overlapped heavily.
  • GRPO, the least diverse arm, scored highest on pass@1.

Evidence

• GSM8K diversity change (first to last checkpoint)
  • Token entropy changed by +0.028 for RFT and −0.055 for GRPO.
  • Unique answers per problem changed by +0.56 for RFT and −1.30 for GRPO.
• Final GSM8K pass@k (3-seed pooled)
  • pass@1 was 0.634 for GRPO, 0.571 for RFT, and 0.574 for RFT+KL.
  • pass@8 was 0.914 for GRPO and 0.910 for RFT, and pass@32 was 0.969 versus 0.972, with overlapping intervals.
• GRPO−RFT pass@1 gap on GSM8K
  • It was +0.058 [0.045, 0.071] against RFT-shortest.
  • The pass@8/32 gaps (+0.005 to +0.020) all included zero.
• MATH-500 levels 4–5 versus the starting model
  • RFT was significantly lower by 3 to 3.5pp at k=1, 2, 4.
  • GRPO showed no significant difference, with the point estimate drifting from −0.7pp at k=1 to −3.7pp at k=16.

How to Apply

• Add the starting checkpoint as a control
  • Run the pre-post-training checkpoint on the same problems and sample count in your eval script.
  • Report pass@k deltas relative to it, not only between trained models.
• Log diversity metrics next to pass@k
  • Sample N completions per problem and compute distinct-3 and answer entropy.
  • Compute them on correct completions only so accuracy does not skew the result.
• Evaluate on a hard subset separately
  • Easy benchmarks saturate near 0.9 by pass@8.
  • Pick low-accuracy problems and compare curves for k from 1 to 16.

Try It

# Run in Python: distinct-3 over correct solutions (a simple version of the paper's metric)

from itertools import islice

def distinct3(texts):
    tri = []
    for t in texts:
        w = t.split()
        tri += list(zip(w, islice(w, 1, None), islice(w, 2, None)))
    return len(set(tri)) / max(len(tri), 1)

correct_solutions = ["{{correct solution 1}}", "{{correct solution 2}}", "{{correct solution 3}}"]
print(distinct3(correct_solutions))

# Feed correct solutions from model A and model B for the same problems and compare the values.
# Next: truncate to equal length and recompute to check for a length effect.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.