When Should LLMs Trust Their Own Revisions? A Risk-Aware Study of Intrinsic Self-Correction

Who Should Read

Backend developer whose automatic second-pass 'double check' step sometimes lowers LLM answer accuracy

Core Mechanics

• Revision can flip correct answers
  • Without external evidence, a second pass fixes some errors but also breaks correct answers.
  • The study covered 29 open-weight models on BoolQ, GSM8K, and Corr2Cause.
• Average accuracy hides the damage
  • Llama-3.1-8B gained 25.5 points on GSM8K yet changed 19.1% of its initially correct answers into wrong ones.
  • Track fixes and breakages separately, not just net accuracy.
• Always revising hurts in many settings
  • Unconditional revision lowered accuracy in 20 of 28 BoolQ, 10 of 28 GSM8K, and 9 of 26 Corr2Cause settings.
  • Pick a policy per model and task.
• Prompt wording changes the outcome
  • On BoolQ with 11 models and five instructions, mean accuracy change was negative for every prompt.
  • The neutral prompt lost least, and forceful switch instructions added more harmful flips.
• A gate that decides whether to revise sometimes helps

Evidence

• Gate reversed the loss for Qwen2.5-7B on BoolQ
  • Unconditional revision dropped accuracy from 80.83% to 72.32%.
  • The gate reached 86.02%, 5.2 points above the initial answer.
• Llama-3.1-8B on GSM8K gains come from revision itself
  • Accuracy went from 17.44% to 42.99% with revision and 43.59% with the gate.
  • The gate added only 0.60 points over revision.
• Gates sometimes missed available gains
  • Qwen3-Coder-30B-A3B on Corr2Cause: 56.06% initial, 83.70% revised, 56.06% gated.
  • DeepSeek-R1-Distill-Qwen-7B: 77.78% revised vs 71.28% gated.
• Llama-3.1-70B on BoolQ broke more than it fixed
  • It recovered 122 errors but overturned 159 correct answers, a net -1.1 points.

How to Apply

• Log correct-to-wrong flips in self-correction
  • Today, store both first and second answers on a small eval set.
  • Count wrong-to-correct and correct-to-wrong separately instead of only net accuracy.
• Remove always-revise as the default
  • Compare keep-initial, always-revise, and gated policies per model and task on your eval set.
  • Ship only the policy with the best accuracy.
• Word revision instructions neutrally
  • Drop forceful phrases like 'you may be wrong, change it'.
  • State explicitly to keep the answer when there is no evidence for a change.
• Build a confidence-based revision gate
  • Extract logprobs, yes/no confidence, and parse success from the first answer.
  • Train a simple classifier on labeled data to predict whether revision is needed.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: revision risk check)

Review the question and first answer below. Do not use any new external information.

Question: {{question}}
First answer: {{first answer}}

Rules:
1. Write CHANGE on the first line only if there is clear evidence of an error; otherwise write KEEP.
2. If CHANGE, give the error evidence and the corrected answer.
3. On the last line, give confidence from 0 to 100.

Next step: run 20 of your own questions, compare KEEP/CHANGE outcomes with ground truth, and count correct-to-wrong flips.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.