Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity? | AI Paper Digest

TL;DR Highlight

Asking a final LLM to judge verifier feedback showed 'previous' label bias, with accuracy swinging from 0.156 to near 1.0 by reasoning effort.

Who Should Read

Backend or agent developers chaining multiple LLMs in draft-verify-revise pipelines who want to know whether the verifier actually catches errors

Core Mechanics

'Previous' points to different values at each stage
- The draft LLM read it as the value in effect before its proposed change (20mg), while the verifier matched the field labeled Previous (15mg).
- The experiment tests whether the final-stage LLM can judge which one is right.
Errors came from label matching
- Models said reverting to the pre-change value was correct in practice, then still picked the value labeled Previous (articulate-then-dismiss).
- Most wrong verdicts followed this pattern, and operational reasoning governed the verdict in very few.
Reasoning effort swings the score hugely
- GPT-5.2 went from 0.156 with no reasoning to 0.942 at xhigh.
- Gemini 3 Pro stayed above 0.94 at every level.
A cheaper setting beat a pricier one
- Gemini 3 Pro at low effort outscored GPT-5.2 at xhigh for roughly 5% of the cost per trial.
- A larger model with too little reasoning can lose to a smaller one (GPT-5.2 none scored below every GPT-5-mini configuration).

Evidence

Top configuration: Gemini 3 Pro low
- Balanced accuracy 0.965, first among independent models in the primary experiment.
- GPT-5.2 xhigh scored 0.942 and Gemini 3 Pro high scored 0.941.
GPT-5.2 went 0.156 to 0.942 with reasoning
- With no reasoning it matched ground truth on only 125 of 600 valid trials (overall accuracy 0.208), below chance.
- Its correct-draft accuracy (specificity) was 0.000.
Cost gaps were large
- GPT-5.2 xhigh cost $0.0556 per trial for 0.942.
- Gemini 3 Flash medium cost $0.0022 for 0.894, and Gemini 3 Pro low cost $0.0030 for 0.965.
Removing labels helped only on opaque stimuli
- The balanced accuracy difference was significant for 4 of 21 configurations, all improvements (GPT-5.2 high +0.115).
- Three condition-level comparisons were significant, all on opaque stimuli and all favoring label removal.

How to Apply

State the reference value in the revise prompt
- Today, search your pipeline prompts for relative expressions like previous, current, or the response.
- Write the intended referent explicitly in each stage's prompt, e.g. 'the value to revert to is the one in effect before the proposed change'.
Try removing label fields from verifier JSON
- If your verifier outputs fields like error_category or risk_level, build a version with only analysis and evidence.
- Run each case 20 times and compare verdict accuracy.
Sweep reasoning effort per model
- Run your final-judge stage across models and effort levels and tabulate accuracy and cost per trial.
- Pick the setting with the best accuracy per dollar, not the top effort level.
Build minimal-pair tests where label and meaning diverge
- Create 10 or more cases where the labeled-field value differs from the operationally correct value.
- Check that a shallow label-matching strategy cannot pass them.

Try It

copy & run

# Paste into Claude or ChatGPT: simplified version of the paper's method (final-stage verdict prompt)

[System instruction]
You are the final judge in a draft-verify-revise pipeline. Decide whether the verifier's feedback is 'correct' or 'incorrect', and give a rationale.

Before judging, write out:
1) What change did the draft propose? (e.g. 20mg→25mg)
2) 'Revert/previous' means not the value in the field labeled Previous, but the value actually in effect right before the proposed change. Compute the revert target yourself using this rule.
3) Compare it with the value the draft cited.

[Input]
System prompt: {{draft LLM system prompt}}
User prompt: {{draft LLM user prompt}}
Draft: {{draft}}
Verifier feedback: {{verifier feedback}}

[Output format]
{"rationale": "...", "score": "correct" | "incorrect"}

Next step: run the same input without rule 2 and compare how the verdict changes.

Terminology

deictic shiftWhen an expression like 'previous', 'now', or 'here', whose meaning depends on the speaker's standpoint, comes to point at a different thing as context passes between pipeline stages. It is like a variable name resolving to a different value after a scope change.

draft-verify-reviseA three-stage pattern where one LLM drafts, a second critiques, and a third revises using the feedback. It resembles write, code review, then fix.

balanced accuracyThe average of accuracy on 'correct' cases and accuracy on 'incorrect' cases, weighted equally. Always giving the same answer scores 0.5.

reasoning effortA provider-specific setting controlling how many thinking tokens a reasoning model spends before answering. More costs more but does not always help.

e-valueA measure of evidence that stays valid even if you stop collecting data at any time. A value of 20 or more rejects the chance hypothesis (α=0.05).

context rotPerformance degradation as a context window fills with too much information, making the model lose focus or ignore instructions.

Related Papers

Related Resources

Original Abstract (Expand)

Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute. One LLM drafts, a second critiques the draft and provides feedback, and a third uses that feedback to revise the draft into the final output. As context cascades between stages, LLMs at different stages can resolve a context-dependent expression such as "previous" differently. When that happens, the expression undergoes a deictic shift, a change in what it refers to. This phenomenon was studied with a synthetic dataset of 10 base examples, each rendered in three conditions. Holding the shared components constant, the conditions varied whether the draft stage LLM (the assistant) or the verify stage LLM (the grader) resolved the expression correctly, and how much independent reasoning the revise stage LLM (the meta-evaluator) needed to determine which reading was correct. Six models from three providers were tested across 21 reasoning effort configurations using e-values for sequential testing, in a primary experiment and an ablation experiment that removed error classification labels from the grader's feedback. A separate LLM analyzed the meta-evaluator's stated rationale for each wrong verdict. Balanced accuracy (the unweighted mean of sensitivity and specificity) ranged from 0.156, below chance, to near-perfect. GPT-5.2 rose from 0.156 without reasoning to 0.942 at its highest reasoning effort level, while Gemini 3 Pro stayed above 0.94 at every level. Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh reasoning effort for roughly 5% of the cost per trial. When the meta-evaluator erred, it tended to rely on surface cues rather than operational reasoning. Context engineers implementing draft-verify-revise pipelines should be wary of deictic shifts and make the intended referent explicit at each stage.