Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models

TL;DR

Auto-RT introduces a reinforcement-learning framework for automatic red-teaming of large language models, addressing sparse and costly exploration by coupling Early-terminated Exploration with Progressive Reward Tracking. By deploying an degraded target model and the First Inverse Rate metric to guide reward shaping, Auto-RT densifies feedback and accelerates discovery of high-exploitability vulnerabilities. Empirical results across 16 white-box and 2 black-box models show Auto-RT achieves higher attack effectiveness, efficiency, and diversity than baselines and remains robust under defenses, approaching or surpassing human-crafted strategies. The work highlights a practical, scalable path for automated vulnerability assessment and alignment optimization in diverse LLM deployments.

Abstract

Automated red-teaming has become a crucial approach for uncovering vulnerabilities in large language models (LLMs). However, most existing methods focus on isolated safety flaws, limiting their ability to adapt to dynamic defenses and uncover complex vulnerabilities efficiently. To address this challenge, we propose Auto-RT, a reinforcement learning framework that automatically explores and optimizes complex attack strategies to effectively uncover security vulnerabilities through malicious queries. Specifically, we introduce two key mechanisms to reduce exploration complexity and improve strategy optimization: 1) Early-terminated Exploration, which accelerate exploration by focusing on high-potential attack strategies; and 2) Progressive Reward Tracking algorithm with intermediate downgrade models, which dynamically refine the search trajectory toward successful vulnerability exploitation. Extensive experiments across diverse LLMs demonstrate that, by significantly improving exploration efficiency and automatically optimizing attack strategies, Auto-RT detects a boarder range of vulnerabilities, achieving a faster detection speed and 16.63\% higher success rates compared to existing methods.

Figures (12)

• Figure 1: Comparison between previous red-teaming approaches and Auto-RT. Previous works focused on identifying safety flaws of the target model under given attack strategies, whereas Auto-RT directly explores systematic safety flaws in target models starting from searching strategies itself, enabling a fully automated process.
• Figure 2: The framework of Auto-RT, comprising two key components: 1) Early-terminated Exploration, which assesses the diversity of the generated strategies and the consistency of the rephrased prompt with the initial toxic behavior to determine the necessity of safety evaluation. If either constraint is unmet, the process immediately terminate, and a penalty is applied; 2) Progressive Reward Tracking, which enhances the density of safety rewards by utilizing a degrade model derived from the target model, thereby improving the efficiency and effectiveness of the exploration process.
• Figure 3: Conceptual diagram of the safety distribution $\mathcal{J}(\textbf{s})$ across the state space $\textbf{s}$, illustrating the principle of our proposed reward shaping process. The red curve represents the safer model $\textcolor{RedOrange}{m}$, while the blue curve represents the less safe model $\textcolor{NavyBlue}{m'}$. $\theta$ denotes the safety-danger threshold, with $\textcolor{RedOrange}{\delta}$ and $\textcolor{NavyBlue}{\delta'}$ marking the respective dangerous subspaces. The safer model, $\textcolor{RedOrange}{m}$, demonstrates higher safety across most states, with its dangerous subspace, $\textcolor{RedOrange}{\delta}$, being sparse and minimally interconnected. In contrast, the less safe model, $\textcolor{NavyBlue}{m{\prime}}$, exhibits larger and more connected dangerous subspaces, increasing the probability of encountering unsafe regions. Notably, the dangerous subspace of $\textcolor{RedOrange}{m}$ is entirely encompassed by that of $\textcolor{NavyBlue}{m{\prime}}$. This relationship allows for the strategic use of $\textcolor{NavyBlue}{m{\prime}}$ to efficiently focus the exploration process on identifying the dangerous subspaces of $\textcolor{RedOrange}{m}$.
• Figure 4: Comparison of attack efficiency between Auto-RT and RL. The violin plots represent the distribution of attack success rates for every 1k sampled strategies, with lighter colors indicating Auto-RT and darker colors representing RL. Auto-RT achieves higher attack success rates than RL under the same number of samples, and with larger variance, indicating that it achieves more comprehensive exploration.
• Figure 5: The relationship between the red-teaming outcomes (Attack ASR) following reward shaping with a series of intermediate models (M1 to M6), the safety levels of these models (Weaken ASR), and their first inverse rate for additional toxic behavior (Weaken FIR). These intermediate models are derived by fine-tuning on six target models using varying amounts of toxic data.The optimal red-teaming results are achieved by selecting the last intermediate model before a sudden spike in FIR (represented by the dark-colored bar in the figure) as the degrade model for reward shaping.