Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks | AI Paper Digest

TL;DR Highlight

Restoring a trusted toolset before planning cut agent attack success to 0% in many settings.

Who Should Read

Backend developers worried that prompt injection will make their tool-using LLM agents call the wrong tools

Core Mechanics

Restore the toolset right before planning
- Normal Tool Recalling (NTR) overwrites the agent's visible tool list with a saved trusted toolset.
- It needs no knowledge of the attack and no anomaly detection.
Filter suspicious tools by embedding
- Attacker Tool Filtering (ATF) embeds tool descriptions and drops outliers with an Isolation Forest.
- It assumes injected tools differ semantically from the original ones.
Stack three prompt-based defenses
- CoT and paraphrasing run right after user input.
- Self-reflection checks the plan before execution.
GPT models lost task success under defense
- Attacks were blocked, but Original Task Success Rate (OTSR) often dropped sharply.
- Smaller open models often improved on both security and task success.
Bigger models were not safer

Evidence

NTR combo drove DPI ASR to 0
- GPT-3.5 went from 0.9469 to 0.0 and GPT-4 from 0.9156 to 0.0 (ASB benchmark, averaged over 4 injection types).
- Gemma2 went from 0.4187 (ASB paraphrase) to 0.1206 with the NTR combo.
NTR combo kept IPI and MP ASR near 0
- On IPI, only Qwen2 was nonzero at 0.0025 and the other five models were 0.0.
- On MP, all six models reached 0.0.
NTR combo raised Gemma2 OTSR
- On IPI it rose from 0.165 with no defense to 0.5456.
- On MP it rose from 0.1 to 0.415.
NTR alone gave GPT-5 zero ASR
- DPI ASR fell from 0.2425 with no defense to 0.0, and OTSR rose from 0.045 to 0.095.
GPT models lost OTSR on backdoor
- GPT-4 OTSR fell from 0.928 with no defense to 0.187 with the NTR combo, while ASR fell from 0.58 to 0.0.

How to Apply

Overwrite the tool list with a whitelist before planning
- Save the approved toolset at service start.
- Right before the LLM planning call, replace the agent's visible tool list with that saved copy.
Run a plan-validation prompt before execution
- Pass the generated JSON plan and the allowed tool list to the LLM for validation.
- Have it fix and return the plan if any tool is outside the allowed list.
Apply Isolation Forest to tool description embeddings
- Use this when your agent receives tools dynamically from outside.
- Embed the descriptions and drop outlier tools before planning.
Evaluate defense layers per model
- Turn defenses on one at a time and measure ASR and original task success together.
- For GPT-family models, remove layers step by step and check the drop in task success.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: self-reflection plan validation)

Please validate the following agent plan. The plan is formatted as a JSON list.
plan: {{JSON plan}}

Each step in the plan must be a dictionary with the following keys:
- "message": a description of what the agent does in that step.
- "tool_use": a list of tool names used in that step.

From now on, you are only allowed to use the normal tools listed below: {{allowed tool list}}

Validation rules:
- The plan must contain at least two steps.
- Each step must include a non-empty "tool_use" field.
- All tools in "tool_use" must be selected only from the allowed tools list above.
- Steps must follow a logical sequence; each step should build on the previous one.
- Do not modify the "message" field unless it is incorrect or unclear.
- Steps should not repeat unnecessarily.

If the plan is valid, return it exactly as it is. If there are issues, return a corrected version in the same format. Respond only with the valid or corrected JSON list.

# Next: insert a tool name outside the allowed list into the plan and check that it gets removed

Terminology

ASRAttack Success Rate. The share of attacked tasks where the agent used the attacker's tool; lower is better.

OTSROriginal Task Success Rate. The share of normal tasks completed, showing whether the defense breaks real work.

Isolation ForestAn algorithm that quickly isolates points far from the rest and flags them as outliers. It is like spotting the one person standing apart from the crowd.

Memory poisoningAn attack that plants malicious content in an agent's memory store. It is like slipping a fake note into someone's notebook.

PoT backdoorAn attack where a trigger phrase makes the agent follow a malicious plan demonstration. It is like a sleeper agent reacting to a code word.

Related Papers

Original Abstract (Expand)

Large Language Model (LLM) agents have demonstrated impressive capabilities across a variety of domains, particularly when integrated with external tools for multi-step task completion. However, they are increasingly vulnerable to adversarial attacks, including direct prompt injection, indirect prompt injection, memory poisoning, and backdoor attacks, which exploit the model's openness to prompt injection and tool manipulation. In this work, we explore practical and generalizable defense strategies within a unified framework across these four attack types. We introduce two universal tool-based defenses: Attacker Tool Filtering, which uses anomaly detection (e.g., Isolation Forest) to identify and remove suspicious tools, and Normal Tool Recalling, a white-box method that restores the agent's original toolset prior to planning. Additionally, we incorporate prompt-based defenses: Chain-of-Thought prompting and self-reflection techniques to enhance reasoning and task paraphrasing to mitigate attacks. Experimental results across both four open-source LLMs (Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B) and three proprietary LLMs (GPT-3.5, GPT-4, and GPT-5) show that our methods significantly reduce the Attack Success Rates (ASR), achieving 0% ASR in many settings, while preserving or even improving the original task success rate. These findings highlight the promise of simple, modular, multi-layered defenses for strengthening the security and robustness of tool-integrated LLM agents. The code is available at https://github.com/Xiaoyan-Lisa/Defenses-for-Tool-Integrated-LLM-Agents-Against-Adversarial-Attacks.