RL-MTJail: Reinforcement Learning for Automated Black-Box Multi-Turn Jailbreaking of Large Language Models

TL;DR

This work tackles the vulnerability of black-box LLMs to multi-turn jailbreaks by reframing attacker training as a trajectory-level reinforcement learning problem. It introduces RL-MTJail, which adds two heuristic process rewards—over-harm mitigation and target-guided progression—to mitigate sparse supervision and promote long-horizon strategies. Empirical results across HarmBench, StrongREJECT, and JailbreakBench show substantial improvements in attack success rates and robust transferability across victim models, with ablation confirming the value of the process rewards. The approach advances understanding of jailbreak dynamics and offers a framework for developing and evaluating defenses against trajectory-based prompts in real-world LLM deployments.

Abstract

Large language models are vulnerable to jailbreak attacks, threatening their safe deployment in real-world applications. This paper studies black-box multi-turn jailbreaks, aiming to train attacker LLMs to elicit harmful content from black-box models through a sequence of prompt-output interactions. Existing approaches typically rely on single turn optimization, which is insufficient for learning long-term attack strategies. To bridge this gap, we formulate the problem as a multi-turn reinforcement learning task, directly optimizing the harmfulness of the final-turn output as the outcome reward. To mitigate sparse supervision and promote long-term attack strategies, we propose two heuristic process rewards: (1) controlling the harmfulness of intermediate outputs to prevent triggering the black-box model's rejection mechanisms, and (2) maintaining the semantic relevance of intermediate outputs to avoid drifting into irrelevant content. Experimental results on multiple benchmarks show consistently improved attack success rates across multiple models, highlighting the effectiveness of our approach. The code is available at https://github.com/xxiqiao/RL-MTJail. Warning: This paper contains examples of harmful content.

Figures (15)

• Figure 1: Illustration of turn-level versus trajectory-level optimization in multi-turn jailbreak attacks. (a) Turn-level optimization greedily selects the query that elicits the most harmful response. (b) In contrast, trajectory-level optimization leverages outcome rewards and auxiliary heuristics to jointly optimize the full interaction.
• Figure 2: An illustrative interaction trajectory demonstrating the deficiency of turn-level optimization. The example highlights an intermediate query that is critical for successfully eliciting the final harmful response, despite receiving a low score (in blue). Harmfulness is evaluated per turn by GPT-4o, where a score of 5 denotes a successful jailbreak (in red).
• Figure 3: Impact of inserted query harmfulness on average trajectory harmfulness. AHS is computed over the interactions.
• Figure 4: Semantic similarity progression in multi-turn jailbreaking trajectories. Dashed lines indicate samples that fail to elicit a harmful response within the maximum of 5 turns, while N denotes the turn at which a sample achieves a successful attack.
• Figure 5: Effect of turn limit on ASR@1. Increasing the maximum number of turns consistently enhances the effectiveness of multi-turn jailbreaks. We exclude Gemma-2-9B-Instruct from this analysis due to its limited context length.