Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks

TL;DR

This work reframes multi-turn red-teaming of LLMs as a goal-directed sequential decision-making problem and introduces DialTree-RPO, an on-policy reinforcement learning framework augmented with dialogue tree rollout, pruning, and adaptive masking. The method combines a two-stage training regime (SFT followed by RL with a GRPO-based objective) and a bespoke reward function derived from a safety guardrail to efficiently discover diverse, long-horizon attack strategies. Empirical results across 10 target models show DialTree-RPO achieves an average attack success rate around 85% and outperforms existing baselines by roughly 25–27 percentage points, with strong transferability to larger models. These findings highlight substantial vulnerabilities of current LLMs in multi-turn settings and demonstrate a scalable, automated approach for stress-testing safety, with implications for defense and broader multi-turn strategic reasoning tasks.

Abstract

Despite recent rapid progress in AI safety, current large language models remain vulnerable to adversarial attacks in multi-turn interaction settings, where attackers strategically adapt their prompts across conversation turns and pose a more critical yet realistic challenge. Existing approaches that discover safety vulnerabilities either rely on manual red-teaming with human experts or employ automated methods using pre-defined templates and human-curated attack data, with most focusing on single-turn attacks. However, these methods did not explore the vast space of possible multi-turn attacks, failing to consider novel attack trajectories that emerge from complex dialogue dynamics and strategic conversation planning. This gap is particularly critical given recent findings that LLMs exhibit significantly higher vulnerability to multi-turn attacks compared to single-turn attacks. We propose DialTree-RPO, an on-policy reinforcement learning framework integrated with tree search that autonomously discovers diverse multi-turn attack strategies by treating the dialogue as a sequential decision-making problem, enabling systematic exploration without manually curated data. Through extensive experiments, our approach not only achieves more than 25.9% higher ASR across 10 target models compared to previous state-of-the-art approaches, but also effectively uncovers new attack strategies by learning optimal dialogue policies that maximize attack success across multiple turns.

Figures (6)

• Figure 1: Illustration of dialogue tree expansion with pruning.(a) Each node $s_{t,k}$ denotes a state defined by the goal and dialogue histories at the $k$-th action branch at turn $t$. Starting from $s_0$, the attacker explores multiple conversation paths across turns, until the target is jailbroken or the maximum turn limit $T_{\max}$ is reached. Malformed or off-topic branches are pruned at each turn. (b) At each state, the attacker generates $n$ candidate actions consisting of a CoT and query. Each query is sent to the target to elicit a response, resulting in a new state. (c) We collect the trajectories that are not pruned from the rollout tree and assign rewards to each trajectory based on whether the target model is jailbroken or not. We set the branching factor $n=2$ and $T_{\max}=3$ for this figure.
• Figure 2: Pre-pruning malformed output rate and reward under three masking schemes.Malformed turns are invalid utterances among unpruned candidates, while malformed trajectories refer to the dialogues containing any malformed turn. A higher malformed ratio indicates that more rollouts are pruned before optimization, resulting in lower training efficiency and greater instability. Our adaptive masking improves training stability significantly by mitigating format unlearning (§\ref{['sec:method_mask']}), preventing training collapse, and enabling a steady reward increase.
• Figure 3: Impact of (a) tree depth, (b) branching width, and (c) group size on ASR (%). Attack success rate generally improves with increased turn limits, branching factors, and group sizes.
• Figure 4: DialTree-RPO has the highest attack success rate while being the most query-efficient.
• Figure 5: Distribution of successful attacks (Score 5) across turns. Successes occur more frequently in later turns, showing the effectiveness of multi-turn strategies.