When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation

Who Should Read

Developer of a code-assistant who wants the LLM to automatically regenerate when it produces wrong code.

Core Mechanics

• Regenerating on uncertainty scores made things worse
  • Uncertainty-based self-correction lowered Pass@1 in 5 of 6 configurations.
  • Drops ranged from -3pp to -10pp.
• Cheap uncertainty signals barely tracked correctness
  • Mean token entropy, verbalized confidence, and semantic entropy probes all showed weak correlation.
  • Only multi-sample P(True) correlated strongly, but it needs K rollouts and is expensive.
• Only test-based regeneration reliably helped
  • Regenerating up to 5 times until tests pass raised Pass@1 in every configuration.
  • Gains were +6 to +26pp on HumanEval and +8 to +20pp on BigCodeBench.
• Weaker models gained more from verification
  • DeepSeek-Coder-1.3B gained the most on both benchmarks.
  • Stronger models already pass most tasks, leaving less headroom.
• Adaptive decoding added cost without gains

Evidence

• Uncertainty regeneration dropped on every HumanEval run
  • All three models fell by -3.23pp to -9.68pp versus greedy (n=31).
  • Llama-3.2-3B went from 0.39 to 0.29 at 4.6× latency.
• HumanEval verification gains
  • DeepSeek-Coder-1.3B went 0.61→0.87 (+25.81pp) at 2.4× latency.
  • Qwen2.5-Coder-3B went 0.81→0.87 (+6.45pp) at 1.9× latency.
• Same pattern on BigCodeBench
  • Verification gave Llama 0.271→0.368, Qwen 0.297→0.374, DeepSeek 0.181→0.381 (n=155).
  • Latency was 5.0× to 5.9×.
• Pearson correlation with correctness (HumanEval)
  • P(True) scored 0.842 for Llama, 0.782 for Qwen, and 0.303 for DeepSeek.
  • Semantic entropy probes were weak at -0.196 to -0.475.

How to Apply

• Switch the retry trigger to test results
  • If your pipeline retries on a confidence score, replace it with unit test pass/fail.
  • Regenerate up to 5 times on failure.
• A/B test score-only retries
  • Compare no retry, score-based retry, and test-based retry on Pass@1 using your internal eval set.
  • Turn off score-based retries if they fall below the baseline.
• Use uncertainty only as a gate for verification
  • In latency-sensitive services, run the test loop only for high-uncertainty requests.
  • The paper did not test this combination, so validate on a small eval set first.
• Measure your latency budget first
  • Verification regeneration took 1.9× to 5.9× the baseline time.
  • Check that this fits your allowed latency.

Try It

# Paste into Claude Code or ChatGPT (simplified version of the paper's method: regenerate up to 5 times until tests pass)

Write Python code for the function spec below, using the tests.

[Spec]
{{function spec}}

[Tests]
{{unit test code}}

Rules:
1. Write the code and actually run the tests.
2. If they fail, read the failure message and rewrite from scratch. Try up to 5 times.
3. Do not state a confidence score. Only test pass/fail counts.
4. At the end, report the number of attempts and pass status in one line.

Next step: have it generate once without tests and compare the results.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.