Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models | AI Paper Digest

TL;DR Highlight

Re-sampling without showing the failed code beat self-repair on accuracy and cost in small code models.

Who Should Read

Backend developers building code agents on small open models (1.5B–7B) that auto-retry after failed tests

Core Mechanics

Showing the failed code hurts
- At 1.5B and 3B, showing the model its own failed program gave lower accuracy than simply re-asking the original prompt.
- At 7B the gap closed to a statistical tie.
Test output added no information
- Real execution feedback was compared with a content-free "wrong" notice (placebo), and the two did not differ.
- The gaps were +0.000, +0.026 and +0.008, none significant.
The cause is anchoring
- Seeing its own code pushes the model toward small edits of it.
- Near-identical retries were 2–14% under blind resampling versus 33–68% when the model saw its own code.
Prompt length is not the cause
- Prepending successful solutions to other tasks changed nothing (within ±3.5 points).
- The harm comes from seeing its own failed attempt, not from longer context.
Reflection is expensive

Evidence

MBPP+ pass@1 at k=8
- 1.5B: resample 0.749, placebo 0.688, feedback 0.688, reflect 0.688.
- 3B: resample 0.778, placebo 0.709, feedback 0.735, reflect 0.743.
- 7B: resample 0.820, placebo 0.802, feedback 0.810, reflect 0.828.
Blind resampling used the fewest tokens
- 1.5B output tokens: resample 63k, placebo 90k, feedback 129k, reflect 345k.
- 3B: resample 76k, reflect 284k. 7B: resample 49k, reflect 122k.
Loss from seeing own output (vs. resample)
- placebo−resample: −0.061 at 1.5B (p=0.006), −0.069 at 3B (p<0.001).
- Against a no-retry baseline feedback looks like +12.2 points at 1.5B, but against blind resampling it loses.
Replication (DeepSeek-Coder, FP16)
- DeepSeek 1.3B: placebo−resample −0.138, feedback−resample −0.114 (both p<10^-4).
- At FP16 1.5B, feedback−resample was −0.063 (p=0.0008), matching −0.061 at Q4.
- Across six configurations, baseline quality correlated with penalty size at r=0.96.

How to Apply

Add a blind-resample control to your retry loop
- Next to your self-repair loop, add a branch that re-samples the original prompt only, at temperature 0.8.
- Compare pass rate and token count on the same task set.
Drop the failed code from retry prompts on small models
- For models under 7B, resend only the original instruction on retry.
- Keep temperature above 0, since greedy decoding returns the same code every time.
Cap retries at about two
- Lower your max iteration count to around 2.
- Most gains come in the first two attempts and the rest only burns tokens.
Treat execution feedback and reflection as costs
- A/B the version that includes test output against one that only says "wrong", at the same budget.
- If there is no difference, remove the feedback step.

Try It

copy & run

# Prompt to paste into Claude or ChatGPT — a simplified version of the paper's method (blind resampling)

# Step 1: Run this in a fresh chat and save the result. Do not continue in the same chat; open a new one each time.
Write the following function in Python. Output code only.
{{function spec}}

# Step 2: If the tests fail, do not paste the failed code. Send the same Step 1 prompt again in a new chat.

# Control for comparison: in the same chat, show the failed code and ask for a fix.
The code below is wrong.
{{failed code}}
Fix it.

# Next: try 10–20 problems and compare pass rates and code similarity between the two approaches.

Terminology

Self-repairReturning failed code and its test results to the model and asking it to fix them. It is like handing back a graded exam and asking for another try.

Blind resamplingAsking the model to solve the original problem again without telling it about the failure. It is like starting over on a blank answer sheet.

Placebo conditionA control that gives the failed code plus only a "this is wrong" notice. It isolates the effect of real information like test output.

AnchoringAfter seeing a previous answer, the model stays near it. Like seeing a wrong answer sheet and being unable to think of a different approach.

pass@1The fraction of tasks that pass tests on one attempt. Here a task counts as solved if any of up to k attempts passes.

MBPP+The MBPP Python function benchmark with many more tests added. This paper used 378 tasks.

Related Papers

Related Resources

https://github.com/vermayuvraj/self-improving-agent

Original Abstract (Expand)

Self-repair - returning a failed program to the model together with its test output and asking for a correction - is a standard component of code agents, and is almost always evaluated against a baseline that does not retry at all. We argue that this comparison confounds the value of the feedback with the value of the extra attempt. Using a placebo-controlled design on MBPP+ at three model scales (1.5B, 3B, 7B), we compare four matched-budget retry conditions: blind resampling, a content-free failure notice, genuine execution feedback, and feedback augmented with verbal self-reflection. Blind resampling is the strongest condition below 7B, and remains statistically tied with the best condition at 7B, while consuming 2.5-5.5x fewer tokens; conditioning on the model's own failed attempt costs 6.1 points at 1.5B (p=0.006), and the informational content of execution feedback adds nothing measurable over the placebo. We attribute this to anchoring: when shown its previous attempt, a model reproduces a near-identical program in 33-68% of retries, against 2-14% under blind resampling. Two further experiments delimit the effect. Retrieved solutions to other tasks change nothing (bounded to +/-3.5 points), which localizes the harm to self-conditioning rather than context length; and reflection, the only condition that measurably weakens the anchor, remains dominated on cost. Replication rules out two competing explanations: the penalty is unchanged at full precision, and it reproduces on an independent model family. Across six configurations spanning two families and two precisions, its magnitude is predicted by baseline quality alone (r=0.96) - the cost of anchoring is the cost of committing to a bad first attempt.