Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents

TL;DR

This work investigates the security of LLM agents that rely on external tools by evaluating eight defenses against indirect prompt injection (IPI) attacks. It introduces adaptive attack techniques—rooted in jailbreak-style methods like GCG variants and AutoDAN—to craft adversarial content that defeats each defense, demonstrating attack success rates consistently above 50%. The findings reveal significant vulnerabilities across detection, input-level, and model-level defenses, with adaptive attacks capable of bypassing protections while revealing trade-offs in valid-output rates. The study underscores the importance of incorporating adaptive threat modeling into defense design and evaluation to ensure robust, reliable LLM agent safety in real-world deployments.

Abstract

Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments. However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks. Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks. In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%. This reveals critical vulnerabilities in current defenses. Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability. The code is available at https://github.com/uiuc-kang-lab/AdaptiveAttackAgent.

Figures (5)

• Figure 1: Defenses and adaptive attacks of indirect prompt injection attacks of LLM agent.
• Figure 2: ASRs (%) for different defenses (in green), and ASRs after implementing adaptive attacks (in red) for both Vicuna-7B based prompted agent and Llama3-8B based finetuned agent. We also display the ASR of the original attacks without any defense or adaptive attacks (in gray).
• Figure 3: Target rates (%) and ASRs (%) after using adaptive attacks for each defense.
• Figure 4: Valid rates for different defenses (in yellow), and valid rates after implementing adaptive attacks (in blue) for Vicuna-7B based agent. We also display the valid rates of the original attacks without any defense or adaptive attacks (in gray).
• Figure 5: Cross-evaluation of attacks and defenses. Each grid represents the ASR of adversarial strings trained under specific adaptive attacks against certain defenses. (a) shows results for the Vicuna-7B agent, while (b) corresponds to the Llama3-8B agent. "None" indicates no defense. "Original" refers to the IPI attacks without adversarial strings, and "None|GCG" denotes the GCG attack for no defense.