Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SafeNeuron: Neuron-Level Safety Alignment for Large Language Models

Zhaoxin Wang, Jiaming Liang, Fengbin Zhu, Weixiang Zhao, Junfeng Fang, Jiayi Ji, Handing Wang, Tat-Seng Chua

TL;DR

Safety alignment in large language models often relies on brittle, behavior-level constraints that concentrate safety in a few parameters. SafeNeuron identifies safety-related neurons using Activation Effect Size and Safety Activation Shift, freezes them, and trains the remaining parameters with a Direct Preference Optimization objective in an iterative regime to build redundant safety pathways. The approach preserves general reasoning abilities while substantially reducing jailbreak ASR and improving robustness to neuron pruning, extending effectively to vision-language models. These results suggest that safe behavior is encoded in distributed internal representations, offering a more resilient and interpretable path for model alignment across modalities.

Abstract

Large language models (LLMs) and multimodal LLMs are typically safety-aligned before release to prevent harmful content generation. However, recent studies show that safety behaviors are concentrated in a small subset of parameters, making alignment brittle and easily bypassed through neuron-level attacks. Moreover, most existing alignment methods operate at the behavioral level, offering limited control over the model's internal safety mechanisms. In this work, we propose SafeNeuron, a neuron-level safety alignment framework that improves robustness by redistributing safety representations across the network. SafeNeuron first identifies safety-related neurons, then freezes these neurons during preference optimization to prevent reliance on sparse safety pathways and force the model to construct redundant safety representations. Extensive experiments across models and modalities demonstrate that SafeNeuron significantly improves robustness against neuron pruning attacks, reduces the risk of open-source models being repurposed as red-team generators, and preserves general capabilities. Furthermore, our layer-wise analysis reveals that safety behaviors are governed by stable and shared internal representations. Overall, SafeNeuron provides an interpretable and robust perspective for model alignment.

SafeNeuron: Neuron-Level Safety Alignment for Large Language Models

TL;DR

Safety alignment in large language models often relies on brittle, behavior-level constraints that concentrate safety in a few parameters. SafeNeuron identifies safety-related neurons using Activation Effect Size and Safety Activation Shift, freezes them, and trains the remaining parameters with a Direct Preference Optimization objective in an iterative regime to build redundant safety pathways. The approach preserves general reasoning abilities while substantially reducing jailbreak ASR and improving robustness to neuron pruning, extending effectively to vision-language models. These results suggest that safe behavior is encoded in distributed internal representations, offering a more resilient and interpretable path for model alignment across modalities.

Abstract

Large language models (LLMs) and multimodal LLMs are typically safety-aligned before release to prevent harmful content generation. However, recent studies show that safety behaviors are concentrated in a small subset of parameters, making alignment brittle and easily bypassed through neuron-level attacks. Moreover, most existing alignment methods operate at the behavioral level, offering limited control over the model's internal safety mechanisms. In this work, we propose SafeNeuron, a neuron-level safety alignment framework that improves robustness by redistributing safety representations across the network. SafeNeuron first identifies safety-related neurons, then freezes these neurons during preference optimization to prevent reliance on sparse safety pathways and force the model to construct redundant safety representations. Extensive experiments across models and modalities demonstrate that SafeNeuron significantly improves robustness against neuron pruning attacks, reduces the risk of open-source models being repurposed as red-team generators, and preserves general capabilities. Furthermore, our layer-wise analysis reveals that safety behaviors are governed by stable and shared internal representations. Overall, SafeNeuron provides an interpretable and robust perspective for model alignment.
Paper Structure (41 sections, 1 theorem, 37 equations, 9 figures, 3 tables, 1 algorithm)

This paper contains 41 sections, 1 theorem, 37 equations, 9 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Alignment and Jailbreak
  4. Safety Neuron
  5. Method
  6. Neuron Representation in Models
  7. Safety Neuron Identification
  8. Activation Effect Size
  9. Directional Safety Activation Shift
  10. Safety Neuron Redundancy Construction
  11. Experimental Results
  12. Experimental Settings
  13. Models and Datasets
  14. Evaluation Metrics
  15. Compared Methods
  16. ...and 26 more sections

Key Result

Theorem 3.1

Fix a layer $l$ and neuron $j$. Assume that activations under unsafe and safe inputs are independently sampled from distributions $A_j^{u}$ and $A_j^{s}$ with finite second moments. Let $T_j$ be the activation effect score defined in Eq. (5), and let $\tau_{\text{ES}}>0$ be a fixed selection thresho where $\Phi(\cdot)$ denotes the standard normal cumulative distribution function, and the $o(1)$ te

Figures (9)

  • Figure 1: Behavior-level alignment mainly constrains outputs without regularizing internal safety units. Our neuron-level safety alignment encourages distributed and redundant safety mechanisms that preserve general utility.
  • Figure 2: Overview of our proposed framework. We first compute neuron activation statistics. Activation Effect Size to quantify statistical separability and Safety Activation Shift to measure directional activation strength. We then freeze these safety neurons and perform preference optimization on the remaining trainable neurons to force the model to construct redundant safety pathways.
  • Figure 3: Layer-wise composition of safety neurons identified from ten harmful-task categories using LLaMA3-8B.
  • Figure 4: Layer-wise distribution of safety neurons across model scales. For models ranging from 1.5B to 32B. The color intensity represents the fraction of safety neurons within layers.
  • Figure 5: Convergence of the shared safety neuron overlap ratio as the number of harmful tasks increases.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 3.1: False Selection Probability of ES