Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automating Deception: Scalable Multi-Turn LLM Jailbreaks

Adarsh Kumarappan, Ananya Mujoo

TL;DR

This work tackles the scalability gap in defending against multi-turn jailbreaks by introducing an automated FITD-based pipeline that generates 1,500 psychologically grounded scenarios and evaluates seven LLM families under multi-turn and single-turn settings. The methodology comprises three phases: data generation with a 5-turn escalation template, automated cross-model testing, and LLM-based evaluation with human-validated agreement (98.0%); the study reveals pronounced context sensitivity in GPT-family models (ASR up to 32 percentage points with history) versus near-robust performance in Gemini 2.5 Flash and strong but imperfect resistance in Claude 3 Haiku. Key contributions include the scalable attack dataset, cross-architecture safety insights, and mitigation guidance such as pretext stripping to neutralize the FITD approach, underscoring the need for context-aware safety architectures. The findings have practical significance for safety evaluation pipelines and inform defense strategies against narrative-based manipulation in real-world deployments.

Abstract

Multi-turn conversational attacks, which leverage psychological principles like Foot-in-the-Door (FITD), where a small initial request paves the way for a more significant one, to bypass safety alignments, pose a persistent threat to Large Language Models (LLMs). Progress in defending against these attacks is hindered by a reliance on manual, hard-to-scale dataset creation. This paper introduces a novel, automated pipeline for generating large-scale, psychologically-grounded multi-turn jailbreak datasets. We systematically operationalize FITD techniques into reproducible templates, creating a benchmark of 1,500 scenarios across illegal activities and offensive content. We evaluate seven models from three major LLM families under both multi-turn (with history) and single-turn (without history) conditions. Our results reveal stark differences in contextual robustness: models in the GPT family demonstrate a significant vulnerability to conversational history, with Attack Success Rates (ASR) increasing by as much as 32 percentage points. In contrast, Google's Gemini 2.5 Flash exhibits exceptional resilience, proving nearly immune to these attacks, while Anthropic's Claude 3 Haiku shows strong but imperfect resistance. These findings highlight a critical divergence in how current safety architectures handle conversational context and underscore the need for defenses that can resist narrative-based manipulation.

Automating Deception: Scalable Multi-Turn LLM Jailbreaks

TL;DR

This work tackles the scalability gap in defending against multi-turn jailbreaks by introducing an automated FITD-based pipeline that generates 1,500 psychologically grounded scenarios and evaluates seven LLM families under multi-turn and single-turn settings. The methodology comprises three phases: data generation with a 5-turn escalation template, automated cross-model testing, and LLM-based evaluation with human-validated agreement (98.0%); the study reveals pronounced context sensitivity in GPT-family models (ASR up to 32 percentage points with history) versus near-robust performance in Gemini 2.5 Flash and strong but imperfect resistance in Claude 3 Haiku. Key contributions include the scalable attack dataset, cross-architecture safety insights, and mitigation guidance such as pretext stripping to neutralize the FITD approach, underscoring the need for context-aware safety architectures. The findings have practical significance for safety evaluation pipelines and inform defense strategies against narrative-based manipulation in real-world deployments.

Abstract

Multi-turn conversational attacks, which leverage psychological principles like Foot-in-the-Door (FITD), where a small initial request paves the way for a more significant one, to bypass safety alignments, pose a persistent threat to Large Language Models (LLMs). Progress in defending against these attacks is hindered by a reliance on manual, hard-to-scale dataset creation. This paper introduces a novel, automated pipeline for generating large-scale, psychologically-grounded multi-turn jailbreak datasets. We systematically operationalize FITD techniques into reproducible templates, creating a benchmark of 1,500 scenarios across illegal activities and offensive content. We evaluate seven models from three major LLM families under both multi-turn (with history) and single-turn (without history) conditions. Our results reveal stark differences in contextual robustness: models in the GPT family demonstrate a significant vulnerability to conversational history, with Attack Success Rates (ASR) increasing by as much as 32 percentage points. In contrast, Google's Gemini 2.5 Flash exhibits exceptional resilience, proving nearly immune to these attacks, while Anthropic's Claude 3 Haiku shows strong but imperfect resistance. These findings highlight a critical divergence in how current safety architectures handle conversational context and underscore the need for defenses that can resist narrative-based manipulation.

Paper Structure

This paper contains 56 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Phase 1: Psychologically-grounded dataset generation
  5. Phase 2: Automated model testing
  6. Phase 3: LLM-based evaluation with human validation
  7. Experiments
  8. Experimental setup
  9. Example jailbreaks
  10. Successful Jailbreak (GPT-4o Mini).
  11. Failed Jailbreak (Claude 3 Haiku).
  12. Discussion
  13. Mitigation strategies
  14. Conclusion
  15. Dataset generation and validation
  16. ...and 41 more sections

Figures (2)

  • Figure 1: Overview of the three-phase methodology. Phase 1: A generative model (GPT-5) creates datasets based on the FITD principle. Phase 2: Target LLMs are evaluated under multi-turn and single-turn conditions. Phase 3: An LLM Judge classifies responses, with human validation, to calculate the ASR.
  • Figure 2: ASR comparison. (a) The average ASR shows a significant increase with conversational history for the GPT family. (b) A breakdown by attack type reveals this vulnerability is most pronounced for Illegal Activities.