AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration

TL;DR

AutoRedTeamer tackles scalable, evolving red-teaming of LLMs by introducing a two-agent architecture: a Strategy Proposer that continuously discovers new attack vectors from research and a Red Teaming Agent that executes automated testing guided by a memory of attack performance. The framework maintains a lifelong attack library and employs memory-based selection to exploit synergies among attack vectors while integrating emerging threats. In extensive experiments across HarmBench and AIR-Bench, AutoRedTeamer achieves higher attack success rates with lower computational costs and generates test cases with diversity comparable to human-curated benchmarks across 314 risk categories. These results demonstrate a scalable, adaptable approach to evaluating and hardening AI safety in evolving threat landscapes.

Abstract

As large language models (LLMs) become increasingly capable, security and safety evaluation are crucial. While current red teaming approaches have made strides in assessing LLM vulnerabilities, they often rely heavily on human input and lack comprehensive coverage of emerging attack vectors. This paper introduces AutoRedTeamer, a novel framework for fully automated, end-to-end red teaming against LLMs. AutoRedTeamer combines a multi-agent architecture with a memory-guided attack selection mechanism to enable continuous discovery and integration of new attack vectors. The dual-agent framework consists of a red teaming agent that can operate from high-level risk categories alone to generate and execute test cases and a strategy proposer agent that autonomously discovers and implements new attacks by analyzing recent research. This modular design allows AutoRedTeamer to adapt to emerging threats while maintaining strong performance on existing attack vectors. We demonstrate AutoRedTeamer's effectiveness across diverse evaluation settings, achieving 20% higher attack success rates on HarmBench against Llama-3.1-70B while reducing computational costs by 46% compared to existing approaches. AutoRedTeamer also matches the diversity of human-curated benchmarks in generating test cases, providing a comprehensive, scalable, and continuously evolving framework for evaluating the security of AI systems.

Figures (12)

• Figure 1: AutoRedTeamer combines automated red teaming evaluation (top) with lifelong attack integration (bottom). During evaluation, the Risk Analyzer decomposes user inputs into testable components, guiding the Seed Prompt Generator to create diverse test cases. The Strategy Designer selects attacks based on performance metrics in Attack Memory, with results evaluated by an Attack Judge and Relevance Check. In parallel, the Attack Proposer discovers new attack vectors by analyzing research papers, implementing promising candidates after validation, and adding successful ones to the Attack Library to maintain comprehensive coverage.
• Figure 2: Example trajectory of a test case generated and refined by AutoRedTeamer. AutoRedTeamer can discover successful test cases that elicit undesired behavior across various risk categories and models. AutoRedTeamer combines attack vectors, which is more effective than individual attacks. More examples can be found in Sec. \ref{['sec:examples']} in the Appendix.
• Figure 3: ASR across 43 AIR level-3 categories on AIR-Bench zeng2024airbench2024safetybenchmark (top) and AutoRedTeamer (bottom). AIR-Bench test cases cover the AIR categories but are human-curated and static. AutoRedTeamer test cases are more effective and do not require human curation.
• Figure 4: Visualization of final test case embeddings for AIR-Bench, AutoRedTeamer, and PAIR. AutoRedTeamer generates more diverse prompts that cover a wide range of the embedding space, with closer coverage to human prompts.
• Figure 7: ASR for top-10 discovered attacks on HarmBench on Llama-3.1-70B. Combinations are represented by the color of their components and have higher ASR than individual attacks. AutoRedTeamer discovers an attack strategy with 0.21 higher ASR than the best baseline.