Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation
Xuying Li, Zhuo Li, Yuji Kosuga, Yasuhiro Yoshida, Victor Bian
TL;DR
This study reveals a critical vulnerability in RAG-based language agents: a simple adversarial prefix, Ignore the document, can override LLM safeguards and retrieved context, enabling dangerous or unintended outputs. By evaluating 1,134 adversarial prompts across multiple state-of-the-art LLMs and attack strategies, the authors demonstrate consistently high attack success rates, underscoring the fragility of current LLM defenses. The work argues that safety measures at the agent level are insufficient when the LLM core is compromised and outlines a roadmap—encompassing hierarchical instruction processing, context-aware evaluation, multi-layered defenses, human-in-the-loop feedback, and benchmarking standards—for building more resilient systems. The practical impact is a call to action for researchers and practitioners to redesign LLM architectures and defense-in-depth strategies to safely deploy RAG-based agents in real-world settings.
Abstract
AI agents, powered by large language models (LLMs), have transformed human-computer interactions by enabling seamless, natural, and context-aware communication. While these advancements offer immense utility, they also inherit and amplify inherent safety risks such as bias, fairness, hallucinations, privacy breaches, and a lack of transparency. This paper investigates a critical vulnerability: adversarial attacks targeting the LLM core within AI agents. Specifically, we test the hypothesis that a deceptively simple adversarial prefix, such as \textit{Ignore the document}, can compel LLMs to produce dangerous or unintended outputs by bypassing their contextual safeguards. Through experimentation, we demonstrate a high attack success rate (ASR), revealing the fragility of existing LLM defenses. These findings emphasize the urgent need for robust, multi-layered security measures tailored to mitigate vulnerabilities at the LLM level and within broader agent-based architectures.