RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent

TL;DR

This work tackles the challenge of context-specific jailbreaks in large language models by introducing RedAgent, a context-aware, memory-augmented, multi-agent red-teaming system. RedAgent formalizes jailbreak attacks as a set of strategies (jailbreak strategies) and uses a Skill Memory to continuously learn from feedback, enabling adaptive planning and autonomous prompt refinement. Through extensive experiments on both general LLMs and real-world GPT-based applications, RedAgent achieves high jailbreak success rates with minimal queries (often within five) and uncovers numerous vulnerabilities, including 60 severe issues across 60 GPT marketplace apps. The approach demonstrates that context-aware prompts and memory-driven planning significantly improve red-teaming efficiency and coverage, with implications for safer LLM deployment and targeted vulnerability disclosure.

Abstract

Recently, advanced Large Language Models (LLMs) such as GPT-4 have been integrated into many real-world applications like Code Copilot. These applications have significantly expanded the attack surface of LLMs, exposing them to a variety of threats. Among them, jailbreak attacks that induce toxic responses through jailbreak prompts have raised critical safety concerns. To identify these threats, a growing number of red teaming approaches simulate potential adversarial scenarios by crafting jailbreak prompts to test the target LLM. However, existing red teaming methods do not consider the unique vulnerabilities of LLM in different scenarios, making it difficult to adjust the jailbreak prompts to find context-specific vulnerabilities. Meanwhile, these methods are limited to refining jailbreak templates using a few mutation operations, lacking the automation and scalability to adapt to different scenarios. To enable context-aware and efficient red teaming, we abstract and model existing attacks into a coherent concept called "jailbreak strategy" and propose a multi-agent LLM system named RedAgent that leverages these strategies to generate context-aware jailbreak prompts. By self-reflecting on contextual feedback in an additional memory buffer, RedAgent continuously learns how to leverage these strategies to achieve effective jailbreaks in specific contexts. Extensive experiments demonstrate that our system can jailbreak most black-box LLMs in just five queries, improving the efficiency of existing red teaming methods by two times. Additionally, RedAgent can jailbreak customized LLM applications more efficiently. By generating context-aware jailbreak prompts towards applications on GPTs, we discover 60 severe vulnerabilities of these real-world applications with only two queries per vulnerability. We have reported all found issues and communicated with OpenAI and Meta for bug fixes.

Figures (9)

• Figure 1: A real-world example of a comparison between general prompt and context-aware prompt in jailbreaking the custom LLM Math mathsolver under the same malicious goal of making Heroin. The red part highlights the context-aware nature of the prompt.
• Figure 2: The typical pipeline of Red Teaming methods to identify the jailbreak threats of the target LLM via attempts of jailbreak prompts.
• Figure 3: Our RedAgent architecture, consisting of three main stages centered around leveraging Skill Memory, which empowers the planner to craft adaptive attacking plans to autonomously refine the jailbreak prompt. The right part of the figure shows the components of our Skill Memory, how it is updated, and how it empowers the crafting of the attacking plan.
• Figure 4: Top-5 effective strategies for GPT-4-1106-preview, GPT-3.5-turbo-1106, Gemini-Pro, LLaMA-2-7b-chat-hf, and Vicuna-7b-v1.5.
• Figure 5: Heatmap for the frequency of effective strategies across various LLMs.