Outcome Monitors: Recovery Affordances for Silent Tool Failures | AI Paper Digest

TL;DR Highlight

Flagging bad tool results and listing recovery tools lifted ToolMaze completion from 10.9% to 28.1%.

Who Should Read

Agent developers whose agents trust well-formatted but wrong tool outputs, like negative stock counts or cached error pages.

Core Mechanics

Silent failures are the dangerous ones
- Timeouts are visible and agents route around them.
- Cached error pages or negative prices arrive in the expected format and get consumed as fact.
Attach a receipt, don't block
- When a tool result violates an outcome contract, the raw result is kept and an advisory receipt is appended.
- The receipt names the violated property and recovery tools, and no action is restricted.
Contracts are mined automatically
- ToolMaze contracts come from nominal traces, AppWorld from public API schemas, τ-bench from the public database.
- Building needs zero model calls and takes under two seconds.
The recovery-tool list does the work
- Removing it drops performance to baseline, and restoring it brings the gain back.
- Witness detail and timing showed no detectable difference, but those tests can't rule out effects under about 18 points.

Evidence

ToolMaze completion 10.9% to 28.1%
- Four models combined went from 35/320 to 90/320, +17.2 points.
- MiniMax M3, run separately, went from 5/80 to 20/80, +18.75 points.
τ-bench retail +14.0 and +12.0 points
- DeepSeek V4 Flash rose from 41/100 to 55/100, and Pro gained +12.0 points.
- Categorical status faults went from 6/50 to 20/50 (+28.0), while conservation faults stayed at 35/50 (0).
Recovery-tool control
- On 114 pairs, the full receipt beat the version without recovery tools by +11.4 points (p=.028).
- The stripped receipt was +0.9 points over baseline, no detectable gain.
Cost versus a reviewer-loop port
- On 80 workflows, receipts beat the Reinforced Agent port by +11.25 points.
- Receipts added 7% to baseline cost ($0.13), while the reviewer added 498% ($9.24).

How to Apply

Wrap tool calls to add recovery tool names
- In 30 minutes, make your tool wrapper return the raw result plus a JSON list of substitute tools when a result looks wrong.
- Never delete the original result.
Start with five value rules
- Add simple checks: prices and stock positive, status in an allowed set, request args echoed in the response.
- Mine them from public schemas or clean logs, no labels needed.
Build a per-tool substitute map
- Group tools that do the same job ahead of time.
- Only list substitutes the agent can actually call.
Measure clean false-positive rate and fault prevalence
- Before shipping, count how often receipts fire on healthy traffic.
- High false positives can break runs that would have succeeded.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: mimicking a receipt in a prompt)

You are an agent that uses tools. The tool result below failed a contract check.

{"tool_result": {{tool result JSON}},
 "outcome_contract": {
  "status": "inconsistent",
  "violation": {"tool": "{{tool name}}", "property": "{{violated property}}", "path": "{{field}}", "expected": "{{expected}}", "observed": "{{observed}}"},
  "admissible_recovery_tools": [{{substitute tools}}]
 }}

This notice is not binding. All actions remain available. Decide whether to trust the result, retry, or switch to a substitute tool, give a one-line reason, then write your next tool call.

Next, try this: delete only the admissible_recovery_tools line, rerun the same input, and compare the agent's behavior.

Terminology

Outcome contractA rule a tool call's result must satisfy, such as stock being at least zero.

ReceiptAn advisory note appended to a violating result, naming the violated property and available recovery tools.

Recovery affordanceThe list of tools that could be used to recover, like signposts showing the agent alternate routes.

Cross-fittingBuilding rules from traces that exclude the evaluated tasks, like not showing the exam questions beforehand.

Silent failureA wrong result that arrives in a normal format with no error signal.

Related Papers

Original Abstract (Expand)

When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.