Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Surgical Refusal Ablation: Disentangling Safety from Intelligence via Concept-Guided Spectral Cleaning

Tony Cristofano

TL;DR

Surgical Refusal Ablation identifies the failure mode of naïve refusal-vector ablation as a polysemantic interference that entangles refusal with core capabilities and linguistic style. By building a Concept Atom Registry and applying Spectral Residualization, SRA orthogonalizes the refusal signal away from Shield and Confound directions, then uses a rank-one update to suppress the cleaned refusal direction. Across multiple models, SRA achieves substantial reduction in distribution drift and preserves math and code distributions as measured by teacher-forced perplexity, challenging the notion of an unavoidable safety tax. The approach demonstrates that precise, atom-guided editing can realize safer, more robust behavior with minimal impact on nonrefusal capabilities, suggesting a general orthogonality-based principle for behavioral model editing.

Abstract

Safety-aligned language models systematically refuse harmful requests. While activation steering can modulate refusal, ablating the raw "refusal vector" calculated from contrastive harmful and harmless prompts often causes collateral damage and distribution drift. We argue this degradation occurs because the raw vector is polysemantic, entangling the refusal signal with core capability circuits and linguistic style. We introduce Surgical Refusal Ablation (SRA) to distill these steering directions. SRA constructs a registry of independent Concept Atoms representing protected capabilities and stylistic confounds, then uses ridge-regularized spectral residualization to orthogonalize the refusal vector against these directions. This yields a clean refusal direction that targets refusal-relevant structure while minimizing disruption to the model's semantic geometry. Across five models (Qwen3-VL and Ministral series), SRA achieves deep refusal reduction (0-2%) with negligible perplexity impact on Wikitext-2 (mean delta PPL approx. 0.02) and minimal distribution drift. Notably, standard ablation on Qwen3-VL-4B induces severe drift (first-token KL = 2.088), whereas SRA maintains the original distribution (KL = 0.044) while achieving the same 0% refusal rate. Using teacher-forced perplexity on GSM8K and MBPP as a high-resolution capability proxy, we show SRA preserves math and code distributions. These results suggest that common "model damage" is often "Ghost Noise," defined as the spectral bleeding of the dirty refusal direction into capability subspaces.

Surgical Refusal Ablation: Disentangling Safety from Intelligence via Concept-Guided Spectral Cleaning

TL;DR

Surgical Refusal Ablation identifies the failure mode of naïve refusal-vector ablation as a polysemantic interference that entangles refusal with core capabilities and linguistic style. By building a Concept Atom Registry and applying Spectral Residualization, SRA orthogonalizes the refusal signal away from Shield and Confound directions, then uses a rank-one update to suppress the cleaned refusal direction. Across multiple models, SRA achieves substantial reduction in distribution drift and preserves math and code distributions as measured by teacher-forced perplexity, challenging the notion of an unavoidable safety tax. The approach demonstrates that precise, atom-guided editing can realize safer, more robust behavior with minimal impact on nonrefusal capabilities, suggesting a general orthogonality-based principle for behavioral model editing.

Abstract

Safety-aligned language models systematically refuse harmful requests. While activation steering can modulate refusal, ablating the raw "refusal vector" calculated from contrastive harmful and harmless prompts often causes collateral damage and distribution drift. We argue this degradation occurs because the raw vector is polysemantic, entangling the refusal signal with core capability circuits and linguistic style. We introduce Surgical Refusal Ablation (SRA) to distill these steering directions. SRA constructs a registry of independent Concept Atoms representing protected capabilities and stylistic confounds, then uses ridge-regularized spectral residualization to orthogonalize the refusal vector against these directions. This yields a clean refusal direction that targets refusal-relevant structure while minimizing disruption to the model's semantic geometry. Across five models (Qwen3-VL and Ministral series), SRA achieves deep refusal reduction (0-2%) with negligible perplexity impact on Wikitext-2 (mean delta PPL approx. 0.02) and minimal distribution drift. Notably, standard ablation on Qwen3-VL-4B induces severe drift (first-token KL = 2.088), whereas SRA maintains the original distribution (KL = 0.044) while achieving the same 0% refusal rate. Using teacher-forced perplexity on GSM8K and MBPP as a high-resolution capability proxy, we show SRA preserves math and code distributions. These results suggest that common "model damage" is often "Ghost Noise," defined as the spectral bleeding of the dirty refusal direction into capability subspaces.
Paper Structure (42 sections, 12 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 42 sections, 12 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. High-level pipeline.
  3. Related Work
  4. Activation Steering & Refusal
  5. Concept Erasure & Orthogonalization
  6. Weight Editing vs. Vector Quality
  7. Model Editing
  8. Null-Space Projection in Editing (Brief Connection)
  9. Method: Surgical Refusal Ablation
  10. The Polysemantic "Dirty" Vector
  11. The Semantic Atom Registry
  12. Data-light construction.
  13. Spectral Residualization (The Cleaning Step)
  14. Iterative Hard-Negative Refinement
  15. Rank-One Weight Update
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Heatmap across concept atoms and the dirty refusal vector. The dirty vector correlates with multiple semantic/style components, motivating cleaning via residualization.
  • Figure 2: Spectral Breakdown: The Anatomy of a "Dirty" Vector. Standard (dirty) and Surgical (cleaned) vectors projected onto concept atoms. SRA removes "Ghost Noise" components aligned with Shield/Style directions while retaining Target (refusal-relevant) signal.
  • Figure 3: Evolution of Semantic Components During Surgery (example: Qwen3-VL-2B, Layer 25). Target components diminish across passes while Shield components remain comparatively stable, indicating targeted removal.