A Causal Perspective for Enhancing Jailbreak Attack and Defense

TL;DR

The paper tackles the problem of understanding jailbreaking in large language models by proposing Causal Analyst, a framework that integrates LLM-based encoding with graph-based causal discovery to identify direct causal drivers from human-readable prompt features. It builds a large dataset of 35k jailbreak attempts across seven LLMs with 37 features, enabling interpretable causal pathways that link features to jailbreak outcomes. The study identifies direct causal drivers such as Positive Character and Number of Task Steps, and demonstrates two practical applications: a Jailbreaking Enhancer that improves attack effectiveness and a Guardrail Advisor that better extracts true user intent. Extensive experiments validate the robustness of the causal approach, show superior structure learning compared to non-causal baselines, and highlight the framework's potential to enhance LLM safety in real-world deployments.

Abstract

Uncovering the mechanisms behind "jailbreaks" in large language models (LLMs) is crucial for enhancing their safety and reliability, yet these mechanisms remain poorly understood. Existing studies predominantly analyze jailbreak prompts by probing latent representations, often overlooking the causal relationships between interpretable prompt features and jailbreak occurrences. In this work, we propose Causal Analyst, a framework that integrates LLMs into data-driven causal discovery to identify the direct causes of jailbreaks and leverage them for both attack and defense. We introduce a comprehensive dataset comprising 35k jailbreak attempts across seven LLMs, systematically constructed from 100 attack templates and 50 harmful queries, annotated with 37 meticulously designed human-readable prompt features. By jointly training LLM-based prompt encoding and GNN-based causal graph learning, we reconstruct causal pathways linking prompt features to jailbreak responses. Our analysis reveals that specific features, such as "Positive Character" and "Number of Task Steps", act as direct causal drivers of jailbreaks. We demonstrate the practical utility of these insights through two applications: (1) a Jailbreaking Enhancer that targets identified causal features to significantly boost attack success rates on public benchmarks, and (2) a Guardrail Advisor that utilizes the learned causal graph to extract true malicious intent from obfuscated queries. Extensive experiments, including baseline comparisons and causal structure validation, confirm the robustness of our causal analysis and its superiority over non-causal approaches. Our results suggest that analyzing jailbreak features from a causal perspective is an effective and interpretable approach for improving LLM reliability. Our code is available at https://github.com/Master-PLC/Causal-Analyst.

Figures (10)

• Figure 1: General diagram of Causal Analyst.
• Figure 2: Applications of causality.
• Figure 3: Sample distributions of training and testing set.
• Figure 4: Label co-occurrence confusion matrix.
• Figure 5: Prior knowledge.