Selective Steering: Norm-Preserving Control Through Discriminative Layer Selection

TL;DR

Selective Steering addresses the problem of inferring-time control over harmful behaviors in large language models without triggering distribution shift. It introduces a norm-preserving rotation and discriminative layer selection to apply steering only where opposite-signed class signals emerge, ensuring stable and continuous control. The approach is underpinned by theoretical guarantees of norm preservation and empirical validation across nine models from three families, achieving up to 5.5× higher attack success rates with zero perplexity violations and near-full preservation of general capabilities. This yields a principled, efficient framework for controllable, robust LLM behavior modification with practical safety benefits.

Abstract

Despite significant progress in alignment, large language models (LLMs) remain vulnerable to adversarial attacks that elicit harmful behaviors. Activation steering techniques offer a promising inference-time intervention approach, but existing methods suffer from critical limitations: activation addition requires careful coefficient tuning and is sensitive to layer-specific norm variations, while directional ablation provides only binary control. Recent work on Angular Steering introduces continuous control via rotation in a 2D subspace, but its practical implementation violates norm preservation, causing distribution shift and generation collapse, particularly in models below 7B parameters. We propose Selective Steering, which addresses these limitations through two key innovations: (1) a mathematically rigorous norm-preserving rotation formulation that maintains activation distribution integrity, and (2) discriminative layer selection that applies steering only where feature representations exhibit opposite-signed class alignment. Experiments across nine models demonstrate that Selective Steering achieves 5.5x higher attack success rates than prior methods while maintaining zero perplexity violations and approximately 100\% capability retention on standard benchmarks. Our approach provides a principled, efficient framework for controllable and stable LLM behavior modification. Code: https://github.com/knoveleng/steering

Figures (12)

• Figure 1: Selective Steering pipeline. At each layer $k$, we compute projections of positive (red) and negative (blue) class means onto the selected feature direction (red/blue boxes). Steering is applied only at layers where projections have opposite signs (layers $k-2$ and $k+1$), using norm-preserving rotation. Layers with same-sign projections (layer $k-1$) remain unchanged.
• Figure 2: Layer-wise heterogeneity in Qwen2.5-7B-Instruct. (a) Activation norms vary substantially across depth, with rapid growth in early layers and amplification near output. (b) Scalar projections class means onto the selected feature direction reveal progressive emergence of opposite-signed discriminability.
• Figure 3: Perplexity measurements across the full steering circle (0°-360°, 10° intervals) for SAS, AAS, and Selective Steering (SS). Each subplot shows one model's perplexity profile, with the baseline (no steering) shown as a dashed circle. Red stars indicate angles where perplexity exceeds the threshold of 2.0, signaling generation instability or collapse. ActAdd and DirAbl are excluded as they provide only single-point steering rather than continuous angular control.
• Figure 4: Controllability of Selective Steering across rotation angles. Each subplot shows attack success rates (ASR) for four evaluators as a function of steering angle $\theta \in [0°, 360°)$. Radial distance indicates ASR magnitude (0.0--1.0). Most models exhibit a clear peak region (typically 180°--270°) where compliance is maximized, demonstrating smooth behavioral control. Note: Gemma family models show bimodal peaks, suggesting the chosen feature direction may not be optimal for this architecture, highlighting the importance of feature extraction quality (see Section \ref{['sec:limitations']}).
• Figure 5: Layer-wise heterogeneity in gemma-2-2b-it.

Theorems & Definitions (4)

• Proposition 1: Norm Violation in Angular Steering
• Proposition 2: Norm Preservation in Selective Steering
• proof : Proof of Proposition \ref{['prop:angular_violation']}
• proof : Proof of Proposition \ref{['prop:norm_preservation']}