Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction

TL;DR

This work presents HARM, a Holistic Automated Red Teamming framework that tackles LLM safety by combining a fine-grained, top-down risk taxonomy with multi-turn adversarial probing. It introduces a comprehensive taxonomy (71 axes, 274 buckets, 2255 descriptors) and six attack vectors to generate ~128k test cases, enabling broad edge-case coverage. The framework pairs a safety reward model with supervised fine-tuning and rejection sampling to train a human-like red-team agent, improving multi-turn probing efficiency and effectiveness beyond prompting-based baselines. Empirical results show that multi-turn red teaming yields deeper insight into vulnerabilities, that safety rewards generalize to out-of-domain data, and that a detect-then-align approach can improve safety with limited impact on helpfulness. Overall, HARM offers a scalable, data-efficient path to safer alignment of LLMs through targeted, multi-turn evaluation and guided refinement of alignment data.

Abstract

Automated red teaming is an effective method for identifying misaligned behaviors in large language models (LLMs). Existing approaches, however, often focus primarily on improving attack success rates while overlooking the need for comprehensive test case coverage. Additionally, most of these methods are limited to single-turn red teaming, failing to capture the multi-turn dynamics of real-world human-machine interactions. To overcome these limitations, we propose HARM (Holistic Automated Red teaMing), which scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. Experimental results demonstrate that our framework enables a more systematic understanding of model vulnerabilities and offers more targeted guidance for the alignment process.

Figures (10)

• Figure 1: Excerpt from the dialogue between our red team agent and Alpaca alpaca, demonstrating a continuous increase in the harmfulness of Alpaca's responses over multiple rounds.
• Figure 2: The overview of our HARM framework. The red-team agent utilizes top-down generated test cases as opening questions and engages in multiple rounds of dialogue with the target language model, aiming to minimize the safety score of each round of the target LM's responses.
• Figure 3: (a) Masking strategy for supervised fine-tuning of a general assistant. (b) Masking strategy for supervised fine-tuning of our red-team agent.
• Figure 4: Average safety scores for different models across five dialogue rounds.
• Figure 5: Flipping rates of different models under various thresholds, where lower rates indicate better resilience to multi-turn red teaming.