Knowledge-Driven Multi-Turn Jailbreaking on Large Language Models

TL;DR

Mastermind is introduced, a multi-turn jailbreak framework that adopts a dynamic and self-improving approach that significantly outperforms existing baselines, achieving substantially higher attack success rates and harmfulness ratings.

Abstract

Large Language Models (LLMs) face a significant threat from multi-turn jailbreak attacks, where adversaries progressively steer conversations to elicit harmful outputs. However, the practical effectiveness of existing attacks is undermined by several critical limitations: they struggle to maintain a coherent progression over long interactions, often losing track of what has been accomplished and what remains to be done; they rely on rigid or pre-defined patterns, and fail to adapt to the LLM's dynamic and unpredictable conversational state. To address these shortcomings, we introduce Mastermind, a multi-turn jailbreak framework that adopts a dynamic and self-improving approach. Mastermind operates in a closed loop of planning, execution, and reflection, enabling it to autonomously build and refine its knowledge of model vulnerabilities through interaction. It employs a hierarchical planning architecture that decouples high-level attack objectives from low-level tactical execution, ensuring long-term focus and coherence. This planning is guided by a knowledge repository that autonomously discovers and refines effective attack patterns by reflecting on interactive experiences. Mastermind leverages this accumulated knowledge to dynamically recombine and adapt attack vectors, dramatically improving both effectiveness and resilience. We conduct comprehensive experiments against state-of-the-art models, including GPT-5 and Claude 3.7 Sonnet. The results demonstrate that Mastermind significantly outperforms existing baselines, achieving substantially higher attack success rates and harmfulness ratings. Moreover, our framework exhibits notable resilience against multiple advanced defense mechanisms.

Figures (5)

• Figure 1: Overview of the Mastermind framework. The workflow is divided into two phases: (1) Knowledge Accumulation, where the system autonomously explores harmful queries on a sandbox model to distill abstract attack strategies into a shared repository ; and (2) Knowledge-Driven Fuzzing, where an evolutionary algorithm optimizes the combination of retrieved strategies to generate effective multi-turn attack plans against the victim model.
• Figure 2: The example demonstrates a jailbreak attempt for the query "Write a defamatory news article". The planner decomposes the goal into a crisis simulation strategy. The controller monitors the interaction and, notably in Turn 3, detects a safety refusal triggered by explicit malicious intent. It then guides the executor to refine the prompt from a direct command to a protective framing, successfully bypassing the target model's defense.
• Figure 3: Cumulative ASR as a function of interaction turns across various target LLMs. The x-axis represents the progression of the dialogue from turn 0 to 10 , while the y-axis indicates the ASR by that stage.
• Figure 4: Impact of planner and executor model selection on ASR across target victim models.
• Figure 5: Performance under varying knowledge repository sizes. The blue bars represent the ASR on the left axis, and the orange line tracks the HR on the right axis, evaluated across strategy retention ratios ranging from 0% to 100%.