Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations

Who Should Read

A backend developer worried that LLM-repaired code after a runtime or API upgrade ships on typical-input tests alone

Core Mechanics

• Tests passed but results were wrong
  • Some changes keep calls succeeding while the result silently changes.
  • Examples: an enum prints 3 instead of Priority.HIGH, and Buffer.poolSize grows eightfold.
  • With typical inputs only, 0.410 of wrong repairs passed the gate.
• Inputs matter, not assertion type
  • Adding one edge input from the release note dropped the rate to 0.065.
  • False rejection of correct repairs stayed at 0.010.
• An LLM can write the test inputs
  • GPT-5.6-luna, never seeing the program, wrote three inputs and reached 0.070.
  • It costs one model call per task.
  • Two local-model writers failed their preregistered tests.
• Letting the model judge which old outputs still bind backfires
  • Using only inputs the writer flagged legacy-binding gave 0.157, worse.
  • The inputs flagged non-binding were the ones that reveal the change.
• Self-tests helped less
  • Wrong releases fell in every family but not significantly after Holm correction.
  • Small models wrote few valid tests, and those tests echoed their own misreading.
• Hard distractors leave a gap
  • 233 wrong candidates matching the old output on typical inputs were added.
  • After the edge input, 0.168 of them still passed.

Evidence

• One edge input: 0.410→0.065
  • Over the full repair space of 52 test tasks in CPython, Node.js and SQLite.
  • Compared with a gate on typical inputs only.
  • On 15 new clusters it went 0.471→0.061.
• Wrong releases per attempt fell for each model
  • Qwen3-8B 0.233→0.057.
  • gpt-oss-20b 0.181→0.020.
  • qwen3.6-35b-a3b 0.120→0.018.
• Held on a menu of 1491 fragments
  • Edge input went 0.278→0.045.
  • Three blind-writer inputs reached 0.053.
• Models' own tests were weaker
  • Qwen3-8B's unguarded self-tests released 0.240 of wrong procedures over the full repair space.
  • The blind writer released 0.070.

How to Apply

• Add one release-note edge input to every repair PR
  • Find where behavior changed in the upgrade's release notes.
  • Turn that boundary into an input and add it next to the existing tests.
  • Get the expected value by running the old code on the old runtime.
• Generate three edge inputs with a separate LLM
  • Give an isolated call only the requirement and the release note.
  • Do not show it the code or hidden tests.
  • Drop inputs that error or are nondeterministic on the old runtime.
• Don't ask the writer whether old output still applies
  • Do not filter inputs by the model's own labels.
  • Filter by successful, deterministic execution on the old runtime.
• Use a stronger model to write tests for weaker repairers
  • Let a cheaper model do the repair.
  • Have a stronger model write the test inputs.

Try It

# Paste into Claude or ChatGPT as-is (simplified version of the paper's method: blind edge-input writer)

You are a test input writer. Use only the requirement and release note below. Do not look at any code.

[Requirement]
{{requirement}}

[Release note excerpt]
{{release note}}

[Typical input and old output]
{{typical input}}

Instruction: Output up to 3 boundary inputs as JSON where the change in the release note could alter the result. Use the same keys and value types as the typical input. Add one line per input saying which aspect of the change it touches. Do not judge whether the old output is still valid.

Next: run the inputs on the old runtime to get expected values, then compare against the repaired code on the new runtime.

Terminology

Related Papers

• Qwen 3.8 27B on my laptop can interpret model weights, too

  Local Qwen 3.8 27B identified 3 of 4 games from raw network weights alone

• Google will turn off search for schools that reject AI

  Reports say schools turning off AI also lose Google Search, and commenters are split

• The 'Astra can read and write weights to improve Qwen' post is misleading

  Astra's Qwen gain used Python and model runs, not just raw weights

• Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app

  Mo, an AI QA agent that explores your app, found about 269 labeled bugs on QA Bench versus 86 for the next best.

• We're releasing the weights of Mistral Large 4

  Mistral released Large 4 weights, claiming top open-weight scores on coding benchmarks

• Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5

  Gemini 4 Carbon takes #1 on Intelligence Index v5 with 59 and is second in coding agents.

• Gemini 4 Carbon

  Google unveils Gemini 4 Carbon, a cybersecurity-focused model, but access is limited to select cyber defenders.

• Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

  On 50 professional tax tasks, the best AI agent passed only 10.8%