Towards Action Hijacking of Large Language Model-based Agent

TL;DR

The paper identifies vulnerabilities in LLM-based agents that use retrieval and memory (RAG) by introducing AI^2, an attack that extracts action-aware knowledge from an application's database and crafts semantically harmless prompts that assemble harmful action plans. It presents a three-phase pipeline—knowledge extraction, hijacking-prompt design, and application hijacking—evaluated across code generation, medical assistants, and Text2SQL domains, achieving high attack success and bypass rates against safety filters. The work demonstrates that safety filters can be circumvented through knowledge-guided prompt assembly and discusses limitations and defense strategies, including cross-component prompt inspection and stronger safety mechanisms. The findings underscore the need for multi-layered security approaches in real-world LLM-based systems to prevent action hijacking and safeguard critical applications.

Abstract

Recently, applications powered by Large Language Models (LLMs) have made significant strides in tackling complex tasks. By harnessing the advanced reasoning capabilities and extensive knowledge embedded in LLMs, these applications can generate detailed action plans that are subsequently executed by external tools. Furthermore, the integration of retrieval-augmented generation (RAG) enhances performance by incorporating up-to-date, domain-specific knowledge into the planning and execution processes. This approach has seen widespread adoption across various sectors, including healthcare, finance, and software development. Meanwhile, there are also growing concerns regarding the security of LLM-based applications. Researchers have disclosed various attacks, represented by jailbreak and prompt injection, to hijack the output actions of these applications. Existing attacks mainly focus on crafting semantically harmful prompts, and their validity could diminish when security filters are employed. In this paper, we introduce AI$\mathbf{^2}$, a novel attack to manipulate the action plans of LLM-based applications. Different from existing solutions, the innovation of AI$\mathbf{^2}$ lies in leveraging the knowledge from the application's database to facilitate the construction of malicious but semantically-harmless prompts. To this end, it first collects action-aware knowledge from the victim application. Based on such knowledge, the attacker can generate misleading input, which can mislead the LLM to generate harmful action plans, while bypassing possible detection mechanisms easily. Our evaluations on three real-world applications demonstrate the effectiveness of AI$\mathbf{^2}$: it achieves an average attack success rate of 84.30\% with the best of 99.70\%. Besides, it gets an average bypass rate of 92.7\% against common safety filters and 59.45\% against dedicated defense.

Figures (5)

• Figure 1: The workflow of an LLM-based framework (ReAct zhang2024agent) in completing a given task from user's instructions.
• Figure 2: Overall pipeline of our proposed action hijacking attack AI$\mathbf{^2}$. In phase 1 (top part), the attacker performs the knowledge extracting attack to extract action-aware knowledge related to his goal from the application's memory. In phase 2 (bottom left part), the attacker crafts a hijacking prompt by adding an adversarial prefix for the redirection instruction, which can induce the application to retrieve the action-aware knowledge and assemble harmful instructions. In phase 3 (bottom right part), the attacker sends the hijacking prompts to the application, which can successfully circumvent the built-in safety filters and induce the generation of detrimental actions.
• Figure 3: Overview of generating the hijacking prompt. Given a harmful action $A^T$ and extracted knowledge $K_t$, AI$\mathbf{^2}$ first initializes the hijacking prompt with a composition of an assembly instruction and adversarial tokens. Then we employ a shadow retriever and an optimizer to iteratively optimize the adversarial tokens, enhancing the probability that the retriever can extract the action-aware knowledge. Additionally, to address the local optima issue caused by the sparse distribution of text vectors, we utilize a checker that exits the optimization process upon encountering a local optimum.
• Figure 4: Visualization of the latent space. Differences in the mapping across various latent spaces lead to intrinsic variations in the distribution of identical prompts across the sample spaces of different models.
• Figure 5: The impact of extracted knowledge and adversarial attack strength on the success rate of hijacking prompt generation.