Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis

Wang Cai, Yilin Wen, Jinchang Hou, Du Su, Guoqiu Wang, Zhonghou Lv, Chenfu Bao, Yunfang Wu

TL;DR

This work proposes Conflict-Aware Sparse Tuning (CAST), a framework that integrates head-level diagnosis with sparse fine-tuning and offers an interpretable and parameter-efficient approach to improving the safety-utility trade-off.

Abstract

Safety alignment in Large Language Models (LLMs) inherently presents a multi-objective optimization conflict, often accompanied by an unintended degradation of general capabilities. Existing mitigation strategies typically rely on global gradient geometry to resolve these conflicts, yet they overlook Modular Heterogeneity within Transformers, specifically that the functional sensitivity and degree of conflict vary substantially across different attention heads. Such global approaches impose uniform update rules across all parameters, often resulting in suboptimal trade-offs by indiscriminately updating utility sensitive heads that exhibit intense gradient conflicts. To address this limitation, we propose Conflict-Aware Sparse Tuning (CAST), a framework that integrates head-level diagnosis with sparse fine-tuning. CAST first constructs a pre-alignment conflict map by synthesizing Optimization Conflict and Functional Sensitivity, which then guides the selective update of parameters. Experiments reveal that alignment conflicts in LLMs are not uniformly distributed. We find that the drop in general capabilities mainly comes from updating a small group of ``high-conflict'' heads. By simply skipping these heads during training, we significantly reduce this loss without compromising safety, offering an interpretable and parameter-efficient approach to improving the safety-utility trade-off.

Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis

TL;DR

This work proposes Conflict-Aware Sparse Tuning (CAST), a framework that integrates head-level diagnosis with sparse fine-tuning and offers an interpretable and parameter-efficient approach to improving the safety-utility trade-off.

Abstract

Safety alignment in Large Language Models (LLMs) inherently presents a multi-objective optimization conflict, often accompanied by an unintended degradation of general capabilities. Existing mitigation strategies typically rely on global gradient geometry to resolve these conflicts, yet they overlook Modular Heterogeneity within Transformers, specifically that the functional sensitivity and degree of conflict vary substantially across different attention heads. Such global approaches impose uniform update rules across all parameters, often resulting in suboptimal trade-offs by indiscriminately updating utility sensitive heads that exhibit intense gradient conflicts. To address this limitation, we propose Conflict-Aware Sparse Tuning (CAST), a framework that integrates head-level diagnosis with sparse fine-tuning. CAST first constructs a pre-alignment conflict map by synthesizing Optimization Conflict and Functional Sensitivity, which then guides the selective update of parameters. Experiments reveal that alignment conflicts in LLMs are not uniformly distributed. We find that the drop in general capabilities mainly comes from updating a small group of ``high-conflict'' heads. By simply skipping these heads during training, we significantly reduce this loss without compromising safety, offering an interpretable and parameter-efficient approach to improving the safety-utility trade-off.
Paper Structure (62 sections, 6 equations, 7 figures, 11 tables)

This paper contains 62 sections, 6 equations, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mitigating the Alignment-Utility Trade-off.
  4. Mechanistic Interpretability for Safety.
  5. The CAST Framework
  6. Framework Overview
  7. Optimization Conflict: Instantiating $\mathcal{O}(h)$
  8. Decoupled Calibration.
  9. Functional Sensitivity: Instantiating $\mathcal{S}(h)$
  10. Unified Conflict Score and Map
  11. Budget-Matched Safety Alignment
  12. Equal-Size Bucketing.
  13. Controlled Intervention.
  14. Experiments
  15. Experimental Setup
  16. ...and 47 more sections

Figures (7)

  • Figure 1: Structural Localization of Alignment Conflict. (Top) The diagnostic map reveals that conflict is concentrated in specific heads (red) rather than globally diffuse. (Bottom) Intervening on these High-Conflict heads precipitates utility degradation (red path), whereas CAST (blue path) selectively targets Low-Conflict heads to secure safety while preserving general capabilities.
  • Figure 2: Overview of the CAST Framework.(A) Diagnosis: We compute a pre-alignment Conflict Score$C(h)$ by synthesizing Optimization Conflict (gradient adversariality) and Functional Sensitivity (causal load). (B) Distribution: We rank heads by $C(h)$, identifying "Risky Zone" vs. "Safe Zone". (C) Alignment: We validate trade-offs via a Budget-Matched Safety Alignment protocol, sparsely updating specific subsets (e.g., Top-25% vs. Bottom-25%) while freezing others.
  • Figure 3: Safety-Utility Pareto Frontier(SFT). Comparison of general capabilities versus safety performance. CAST-SFT (Safe Zone) dominates the frontier across all models, maintaining high utility while achieving robust safety alignment.
  • Figure 4: Safety-Utility Pareto Frontier. We compare the trade-off between general utility (y-axis) and safety (x-axis). Our method, CAST + PCG (Safe Zone), consistently dominates the frontier across all models, achieving the highest utility while maintaining robust safety performance compared to Full SFT and Full PCG.
  • Figure 5: Internal Validity Check (MMLU-CR). Relationship between predicted conflict intensity $\overline{C}(\mathcal{B}_i)$ (x-axis) and the realized MMLU-CR (y-axis) across four buckets ($\mathcal{B}_1 \sim \mathcal{B}_4$).
  • ...and 2 more figures