SARC-DQ: Runtime Data-Quality Gating for Agentic AI: Silent Evidence Defects, the Incompetence Shield, and Downstream-Only Remediation

Who Should Read

Backend or agent developers worried that their agent will act on stale prices or superseded records and trigger wrong orders or actions.

Core Mechanics

• Models can't see metadata-only defects
  • A 120-day-old price is a perfectly well-formed number, so the payload gives no hint.
  • If freshness, provenance, or version never reach the context, no model size can detect the problem.
• Pricier models get fooled just the same
  • Across haiku→sonnet→opus→fable, inference price rose about 15x, yet the defect-conversion rate went only from 60% to 62%.
  • Doubt-marker AUC stayed at or below 0.50 and the data-quality flag rate was 0% at every tier.
• Better agents are more vulnerable
  • The naive agent (elasticity 0.000) converted 0% of defects, protected only by being bad at the task (the paper calls it the incompetence shield).
  • The competent agent (elasticity 0.992) acted on the stale price 42% of the time.
• A metadata gate right before action catches them
  • The Pre-Action Gate caught freshness and schema violations 100% of the time, while the payload-only critic caught 0%.
  • Bad values are quarantined and replaced from a governed buffer, with no writes to source stores.

Evidence

• Price up 15x, conversion flat
  • Metadata-borne action-defect rate (ADR) was 60% at haiku and 62% at fable.
  • Marker AUC was at most 0.50 and explicit flags were 0% at every tier.
• Gate vs payload-only critic detection
  • stale_master_data: critic 0%, gate 100%. schema_drift: 0% to 100%.
  • missing_mandatory_field: 72% to 100%, with n=100 corrupted episodes per class.
• Overall comparison with a realistic critic
  • Detection was 62% for the gate vs 31% for the critic (2x), with 0% false-blocks for both.
  • Residual loss was 294 for the gate vs 311 for the critic; the confidence intervals overlap.
• Freshness fully recovered, portfolio not
  • stale_master_data loss dropped from 134 ungated to -0.5 gated.
  • Portfolio recovery was -0.04, far below the 0.80 target.
• The analytical model matches measurements

How to Apply

• Log metadata fields into agent inputs
  • Today, check that tool responses and retrieval results carry updated_at, source, and version.
  • Fill in missing fields first, since anything absent from context cannot be detected.
• Add a freshness gate before actions
  • Right before write-type tool calls like orders, payments, or updates, block records older than max_age.
  • On violation, choose one response: block, escalate to a human, or quarantine and substitute.
• Substitute from a buffer, not the source
  • Never edit the source record; swap in a verified value from a versioned buffer.
  • Log the IDs of the records each action relied on for traceability.
• List the defects your gate can't see
  • Write down defect types with no rule yet, such as unit changes and outliers, and add dedicated checks.
  • Don't expect a model upgrade to fix them; track coverage rule by rule.

Try It

# Paste into Claude/ChatGPT - simplified version of the paper's method (mimicking a metadata gate with a prompt)

You are an agent's Pre-Action Gate. Inspect the evidence records below before the action.
Rules: freshness(max_age={{max_days}} days), lineage_present, schema_conformant, complete(required={{required_fields}}).

[Evidence records as JSON, including metadata (updated_at, source, version)]
{{records}}

[Action the agent intends to take]
{{action}}

For each record, output a PASS/FAIL table per rule with reasons. If any FAIL, choose one of block / escalate / quarantine-and-substitute. Finally, list risks these rules cannot check (unit changes, outliers, etc.).

Next step: deliberately set updated_at to 120 days ago and confirm it FAILs.

Terminology

Related Papers

• Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

  Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.

• Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents

  Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.

• Harness Engineering 101: The LLM is the brain, the harness is the body

  An agent is a loop managing a JSON array; 16 chapters build one from scratch

• Harness2 by Google Research – Recursive Agent Harnessing

  Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.

• Ousted Savannah police officer gave ICE agent husband access to Flock system

  A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired

• Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling

  Running Claude Code agents in tmux works well, but limits appear around four agents.