Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unraveling LLM Jailbreaks Through Safety Knowledge Neurons

Chongwen Zhao, Yutong Ke, Kaizhu Huang

TL;DR

The paper targets jailbreak vulnerabilities in large language models by identifying and interpreting safety-related neurons within the MLPs. It introduces a vocabulary-projection interpretability method that reveals a duality: harmful prompts trigger safety-driven rejection while benign prompts trigger conformity, via identified safety neurons. Building on this, two defenses are proposed: ActCali, an embedding-level calibration that causally steers outputs, and SafeTuning, a neuron-focused fine-tuning strategy that reinforces safety-critical activations. Across multiple models and tasks, ActCali achieves near-perfect jailbreak success under calibration, while SafeTuning substantially reduces attack success rates and preserves model utility, offering both mechanistic understanding and practical defense insights for safer LLM deployment.

Abstract

Large Language Models (LLMs) are increasingly attracting attention in various applications. Nonetheless, there is a growing concern as some users attempt to exploit these models for malicious purposes, including the synthesis of controlled substances and the propagation of disinformation, a technique known as "Jailbreak." While some studies have achieved defenses against jailbreak attacks by modifying output distributions or detecting harmful content, the exact rationale still remains elusive. In this work, we present a novel neuron-level interpretability method that focuses on the role of safety-related knowledge neurons. Unlike existing approaches, our method projects the model's internal representation into a more consistent and interpretable vocabulary space. We then show that adjusting the activation of safety-related neurons can effectively control the model's behavior with a mean ASR higher than 97%. Building on this insight, we propose SafeTuning, a fine-tuning strategy that reinforces safety-critical neurons to improve model robustness against jailbreaks. SafeTuning consistently reduces attack success rates across multiple LLMs and outperforms all four baseline defenses. These findings offer a new perspective on understanding and defending against jailbreak attacks.

Unraveling LLM Jailbreaks Through Safety Knowledge Neurons

TL;DR

The paper targets jailbreak vulnerabilities in large language models by identifying and interpreting safety-related neurons within the MLPs. It introduces a vocabulary-projection interpretability method that reveals a duality: harmful prompts trigger safety-driven rejection while benign prompts trigger conformity, via identified safety neurons. Building on this, two defenses are proposed: ActCali, an embedding-level calibration that causally steers outputs, and SafeTuning, a neuron-focused fine-tuning strategy that reinforces safety-critical activations. Across multiple models and tasks, ActCali achieves near-perfect jailbreak success under calibration, while SafeTuning substantially reduces attack success rates and preserves model utility, offering both mechanistic understanding and practical defense insights for safer LLM deployment.

Abstract

Large Language Models (LLMs) are increasingly attracting attention in various applications. Nonetheless, there is a growing concern as some users attempt to exploit these models for malicious purposes, including the synthesis of controlled substances and the propagation of disinformation, a technique known as "Jailbreak." While some studies have achieved defenses against jailbreak attacks by modifying output distributions or detecting harmful content, the exact rationale still remains elusive. In this work, we present a novel neuron-level interpretability method that focuses on the role of safety-related knowledge neurons. Unlike existing approaches, our method projects the model's internal representation into a more consistent and interpretable vocabulary space. We then show that adjusting the activation of safety-related neurons can effectively control the model's behavior with a mean ASR higher than 97%. Building on this insight, we propose SafeTuning, a fine-tuning strategy that reinforces safety-critical neurons to improve model robustness against jailbreaks. SafeTuning consistently reduces attack success rates across multiple LLMs and outperforms all four baseline defenses. These findings offer a new perspective on understanding and defending against jailbreak attacks.

Paper Structure

This paper contains 34 sections, 15 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Identifying Safety Knowledge Neurons
  4. Safety Knowledge Neurons inside LLM
  5. Interpreting Model Safety by Vocabulary
  6. Achieving Jailbreak via Calibrating Safety Knowledge Activations
  7. Experiment Setup
  8. Experiment Results
  9. Defending Jailbreak via Tuning Safety Knowledge Neurons
  10. SafeTuning
  11. Experiment Setup
  12. Experimental Results
  13. Further Analysis
  14. Conclusion
  15. Limitation
  16. ...and 19 more sections

Figures (6)

  • Figure 1: (a) Our interpretation method for safety knowledge. (b) Different activation patterns between harmful and benign prompts. (c) Adjusting responses by adjusting activations.
  • Figure 2: Interpretation of the safety knowledge neuron in Llama-2 on the vocabulary table.
  • Figure 3: Overview of SafeTuning
  • Figure 4: Usefulness Score and ASR as tuning ratio.
  • Figure 5: One possible explanation for the results of the approaches. (a) Other methods are perturbed by other layers' results. (b) Other methods are then perturbed by the decomposition process. Our method directly translates the current layer's gain, resulting in conceptually coherent keywords.
  • ...and 1 more figures