Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

TL;DR

This work tackles automated red-teaming for large language models by factorizing the problem into generating diverse attacker goals and producing effective attacks via reinforcement learning. It introduces auto-generated rule-based rewards (RBRs) and a few-shot similarity mechanism to maintain diversity while achieving high attack success. A multi-step RL framework further enhances style-based diversity by conditioning new attacks on past attempts and projecting attacks into a style subspace. Empirical results on indirect prompt injections and safety jailbreaking demonstrate that the proposed approach yields considerably more diverse and still effective attacks than prior methods, while also highlighting limitations such as reward-hacking risk and evaluation challenges. The method holds practical potential for robust adversarial testing and safety evaluation of AI systems, with avenues for stronger evaluation metrics and reduced training variance in future work.

Abstract

Automated red teaming can discover rare model failures and generate challenging examples that can be used for training or evaluation. However, a core challenge in automated red teaming is ensuring that the attacks are both diverse and effective. Prior methods typically succeed in optimizing either for diversity or for effectiveness, but rarely both. In this paper, we provide methods that enable automated red teaming to generate a large number of diverse and successful attacks. Our approach decomposes the task into two steps: (1) automated methods for generating diverse attack goals and (2) generating effective attacks for those goals. While we provide multiple straightforward methods for generating diverse goals, our key contributions are to train an RL attacker that both follows those goals and generates diverse attacks for those goals. First, we demonstrate that it is easy to use a large language model (LLM) to generate diverse attacker goals with per-goal prompts and rewards, including rule-based rewards (RBRs) to grade whether the attacks are successful for the particular goal. Second, we demonstrate how training the attacker model with multi-step RL, where the model is rewarded for generating attacks that are different from past attempts further increases diversity while remaining effective. We use our approach to generate both prompt injection attacks and prompts that elicit unsafe responses. In both cases, we find that our approach is able to generate highly-effective and considerably more diverse attacks than past general red-teaming approaches.

Figures (9)

• Figure 1: System overview. We describe above the multiple stages of calling multiple models. Note, $g$ are the generated attacker goals, $p$ are the attacker prompts, $y$ is the output of the defender target LLM, and $r$ is the reward for RL.
• Figure 2: Few-shot Reward Generation
• Figure 3: Reward Generation from Data
• Figure 4: Main results for indirect prompt injection: We find that our method is effective in generating diverse (larger cosine similarity) and successful prompt injections, and that the multi-step RL reward improves diversity over steps.
• Figure 5: Main results for safety jailbreaking: Our method can trade-off success rate and diversity, with clear trend where the multi-step approach is able to improve diversity over time.