When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation

Who Should Read

Developers running agent benchmarks or in-house eval pipelines who see unstable scores and want to know why

Core Mechanics

• Stopping is not the same as settled
  • Evals usually score the state when the agent stops.
  • Tool calls may still be running then, so the outcome can change after scoring.
• Finality and separation are independent
  • Outcome finality means later events cannot change the result.
  • Cross-unit separation means earlier runs cannot change later runs' conditions.
  • One can hold while the other fails.
• Shared state lets run A change run B's score
  • A delayed write can overwrite the next run's initial value.
  • Then runs are not separate trials, and metrics like pass^k lose their independence assumption.
• No evidence, no final label
  • If remaining effects could flip the label, report unresolved instead of success or failure.
  • If runs stay connected, model the link or group them as one unit.

Evidence

• Snapshot scoring missed delayed writes
  • In the fixed-trace replay, endpoint and terminal labels differed in all 150 nonzero-delay trials.
  • Snapshot scoring gave 50 of 200 successes; reconciliation gave 200 of 200.
• Verified cancellation turned 150 into failures
  • The 150 pending writes were cancelled and confirmed, and each was scored as failure.
  • The synchronous control had 0 disagreements in 600 trials across three scoring policies.
• Shared state exposed 150 of 200 pairs
  • Namespaced, verified-reset, and no-write control conditions each had 0 of 200 exposures.
  • Run B's mean score was 0.75 lower under shared state than under namespacing.
  • Running B before A cut exposure from 150/200 to 0/200.
• Ten public protocols reviewed
  • Reset or deliberate retention: 8 explicit, 2 partial.
  • Open or descendant work: 0 explicit, 2 partial, 6 not reported, 2 not applicable.
  • Evidence for separate scoring: 3 explicit, 7 partial.

How to Apply

• Add a settle check before scoring
  • Right before scoring, check for leftover processes, queues, and async writes, then wait or cancel-and-confirm.
  • If you cannot confirm, record the case as unresolved rather than success or failure.
• Namespace state per run
  • Prefix shared DBs, services, credentials, and file paths with the run ID.
  • If you rely on reset, read back the initial value to verify it before the run starts.
• Keep an open-effects log
  • In your tool wrappers, record the action, handle, endpoint status, and any cross-run route for each async operation.
  • An empty log is not proof, so add system-level checks too.
• Group runs you cannot isolate
  • Treat connected run pairs as one analysis unit when you compute uncertainty.
  • Do not apply the independent-trial assumption of pass^k to them.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: eval design audit)

I run an LLM agent evaluation pipeline. Based on the description below, audit 'outcome finality' and 'cross-unit separation' separately.

[Eval description]
{{Eval setup: what counts as success, when runs stop, which tools and services are used, what is reset between runs}}

[Request]
1. List operations that may outlast the endpoint (async writes, background processes, queues, callbacks).
2. List resources that may persist across runs (credentials, files, shared DBs, memory, external accounts).
3. For each item, make a table: action / responsible system / handle / endpoint status / possible later states / cross-run route.
4. For each item, say whether there is evidence to fix the label; if not, tell me whether to wait, cancel and confirm, or report unresolved.
5. Suggest one way to verify that each isolation or reset actually worked.

Next step: pick one item with no evidence and add a check for it to your real harness.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.