AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses | AI Paper Digest

TL;DR Highlight

A strong model built harnesses lifting GPT-5.4-mini accuracy from 0.49 to 0.91 without any training.

Who Should Read

LLM app developers using a small, cheap model whose accuracy falls short and who are tuning prompts and pipelines.

Core Mechanics

A strong model builds the harness for a weak one
- With no fine-tuning, a strong builder model writes routing, prompts and code solvers that help a weaker target model.
- The builder sees only 5% of the data (195 items) and iterates on it.
The core move is offloading reasoning to code
- Harnesses that shifted unstable reasoning into deterministic code, added per-benchmark routing and forced answer formats worked best.
- Longer reasoning or more sampling was not the main driver.
Builder quality beats platform choice
- Whether the builder ran in Cursor, Claude Code or GPT Codex made only a small difference.
- Raising builder reasoning effort improved harness quality monotonically.
More validation runs did not help
- The number of validation runs was nearly uncorrelated with final accuracy.
- The 5% validation score closely matched the full-set score.

Evidence

Average accuracy went from 0.49 to 0.91
- Best harness (GPT-5.5 on GPT Codex) scored 0.912 vs 0.488 for unscaffolded GPT-5.4-mini, +0.423.
- This is over four Theory-of-Mind benchmarks and 3900 items; the mean of 57 runs was 0.763 (+0.275), and every run beat the baseline.
It beat a bigger model
- Unscaffolded GPT-5.4 scored 0.619, while the best harnessed GPT-5.4-mini scored 0.912.
- The human-built UserHarness still scored slightly higher at 0.939.
Effort raised quality monotonically
- Opus-4.7 as builder: low 0.711, medium 0.793, high 0.807, x-high 0.856.
- x-high vs high gave p=0.013.
Determinism and accuracy
- The share of items answered by code correlated with accuracy at Pearson r=0.72 (n=18).
- The best harness fixed 1717 baseline errors and broke 105 correct answers (McNemar, 3900 items).

How to Apply

Audit the basics of your weak-model calls today
- Check for temperature 0, forced answer-format parsing and per-type routing; add whatever is missing.
- Nearly all harnesses in the paper (95-100%) used these basics.
Have a strong model write the harness code
- Give it about 5% labeled validation samples and run the strongest model at high reasoning effort to produce a router, parser and rules.
- Limit validation evaluations to around five, and borrow the paper's instruction file format.
Build 2-3 independent harnesses and pick the best
- Build several with the same setup and keep the one with the highest validation score.
- Deterministic-solver approaches swing widely when one rule is wrong, so filter them on validation.
Apply selectively to already-strong models
- Measure baseline accuracy per sub-task first and add rules only where headroom is large.
- Adding rules where the baseline is already high can break correct answers.

Try It

copy & run

# Paste into Claude Code or Claude/ChatGPT (simplified version of the paper's method)

You are a harness builder. Goal: design an inference-time harness that helps a weaker model solve the task below. Do not change model weights.
Task description: {{task description}}
Labeled validation samples: {{20-50 validation samples}}

Rules:
1. Look at the samples, group them by type, and find parts that can be solved deterministically in code.
2. Build per-type routing, prompt templates and a strict answer-format parser.
3. Use temperature 0 and call the model only for items that truly need it.
4. Do not hard-code validation answers.

Output: Python code, the reasoning behind the design, and expected failure types.

Next step: paste the failing samples and have it revise the harness. Limit this to about 5 rounds.

Terminology

Harness / scaffoldThe code around a model: routing, prompt templates, verification and tool calls. It leaves the model unchanged and makes the task easier for it.

Builder / targetThe builder is the strong model that writes the harness; the target is the weaker model that uses it to solve the task.

Theory-of-Mind (ToM)Reasoning about what other people believe and want. The paper evaluates on BigToM, Hi-ToM, MMToM-QA and MuMA-ToM.

HeadroomOne minus baseline accuracy, meaning how many correctable errors are left.

Deterministic offloadingAnswering items directly with code and rules instead of model reasoning, so results are stable and repeatable.

Related Papers

Original Abstract (Expand)

Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.