Rethinking Indirect Prompt Injection as a Test-Time Search Problem | AI Paper Digest

TL;DR Highlight

Giving an attacker agent more tokens kept raising indirect prompt injection success rates.

Who Should Read

A backend developer running tool-using LLM agents who needs to test defenses against prompt injection

Core Mechanics

Attack success depends on attacker budget
- A fixed-budget test can underestimate an agent's vulnerability.
- Success rose as the attacker's token budget grew.
Attacks work right next to the user task
- Injections succeeded when placed in paths the agent already had to read.
- Instructions that looked like a normal next workflow step worked best.
Without strategy tracking, attackers repeat themselves
- Removing the strategy tool caused excess reconnaissance and repeated patterns.
- The full harness changed the logical link between task and attack after failures.
Easy actions get hit almost every time
- One-call actions like flagging a message, deleting a file, or sending email had very high success.
- Goals like posting PII publicly or sending credentials were mostly blocked.
More tools make attacks easier but break the real task

Evidence

Opus-4.8 on Retail still had 20.6% ASR at 500K tokens
- GPT-5.5 began to degrade noticeably at around 50K tokens.
- The two victim models separated clearly on ASR@T curves.
Workspace fell faster than Retail
- With GPT-5.5 at 100K tokens, ASR was 52.1% on Workspace versus 37.5% on Retail.
Easy Workspace goals reached 92.5-97.5% ASR
- File deletion hit 97.5%, email sending and event creation 95.0% each, email body exfiltration 92.5%.
- Sending a Facebook security code was 2.5% (1/40), and 0% when deleting the sent mail too.
PII posted publicly in Retail was 0%
- Simple state changes reached 100%, and internal-note exfiltration was 47.1-76.5%.
- From 250K to 500K tokens, mean GPT-5.5 ASR rose from about 49.0% to about 57.8%.

How to Apply

Add token-budget curves to injection tests
- Run your red-team script with attacker token caps such as 50K, 100K, and 500K.
- Log success rate per budget and don't treat one pass as proof of safety.
Put human approval on high-risk tools
- Start with one-call tools like sending email, payments, and deletion.
- Separately block calls that send sensitive data to external channels.
Remove write tools from read-only tasks
- Narrow the tool list per task type to shrink the attack surface.
- Check whether unnecessary state-changing tools are exposed during normal tasks.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: audit my agent's attack surface)

I run an LLM agent. Using the info below, audit it for indirect prompt injection risk.
User task: {{example user task}}
Agent tool list: {{tool names and descriptions}}
Paths that read external text: {{email, documents, reviews, etc.}}

1. List the paths the agent must read for this task.
2. For each path, classify malicious instruction types that could look like a normal next workflow step.
3. Separate one-call high-risk tools from tools that can leak sensitive data.
4. Suggest one defense for each risk.

Next: apply the defenses and run the same audit again.

Terminology

Indirect prompt injectionAn attack that hides malicious instructions in emails or documents an agent reads. The agent follows them even though the user never asked.

Test-time computeComputation spent at run time, such as thinking or trying longer, instead of in training. Like giving a student more time on an exam.

HarnessThe execution wrapper that lets an agent use tools, keep records, and receive feedback.

ASRAttack success rate: the fraction of task pairs where the attacker achieved its goal.

JSRJoint success rate: the fraction where the attack succeeded and the user task was also completed. Stricter than ASR.

Related Papers

Original Abstract (Expand)

We formulate indirect prompt injection as a test-time search over a task-dependent attack surface induced by the environment, user task, and injection task. To operationalize this formulation, we introduce an agentic attacker with a dedicated search harness that performs environment reconnaissance, structured reasoning over attack strategies, and adaptive evaluation using victim-agent feedback. Across heterogeneous tasks, we find that increasing attacker test-time compute improves vulnerability discovery and exploitation, while ablations show that explicit strategy management is important for avoiding redundant search and sustaining gains at larger budgets. These results suggest that agentic security evaluations should characterize both the attacker's search procedure and compute budget, rather than treating attack success as a budget-independent property of the victim. More broadly, our findings identify the attacker's adaptive search over the system attack surfaces as an important and underexplored security risk for tool-using agents.