Angular Steering: Behavior Control via Rotation in Activation Space

TL;DR

Angular Steering reframes activation editing as a norm-preserving rotation in a fixed $2$-D subspace of activation space, enabling fine-grained, continuous control over model behaviors while maintaining overall language modeling performance. By extracting a target feature direction via a Difference-in-Means approach and constructing a steering plane through PCA, the method generalizes prior activation edits as rotations and introduces an adaptive variant that limits steering to positively aligned activations. Empirical results across multiple model families show robust control of refusal-like behaviors with minimal degradation to baseline capabilities, and the adaptive variant improves coherence, particularly on smaller models where feature interference is more pronounced. The approach unifies existing steering methods under a geometric framework, offering interpretable, scalable behavior modulation for safer deployment of LLMs. Practical implementation details, including efficient rotation formulas and evaluation on diverse benchmarks, support real-world applicability and future extensions to broader alignment goals.

Abstract

Controlling specific behaviors in large language models while preserving their general capabilities is a central challenge for safe and reliable artificial intelligence deployment. Current steering methods, such as vector addition and directional ablation, are constrained within a two-dimensional subspace defined by the activation and feature direction, making them sensitive to chosen parameters and potentially affecting unrelated features due to unintended interactions in activation space. We introduce Angular Steering, a novel and flexible method for behavior modulation that operates by rotating activations within a fixed two-dimensional subspace. By formulating steering as a geometric rotation toward or away from a target behavior direction, Angular Steering provides continuous, fine-grained control over behaviors such as refusal and compliance. We demonstrate this method using refusal steering emotion steering as use cases. Additionally, we propose Adaptive Angular Steering, a selective variant that rotates only activations aligned with the target feature, further enhancing stability and coherence. Angular Steering generalizes existing addition and orthogonalization techniques under a unified geometric rotation framework, simplifying parameter selection and maintaining model stability across a broader range of adjustments. Experiments across multiple model families and sizes show that Angular Steering achieves robust behavioral control while maintaining general language modeling performance, underscoring its flexibility, generalization, and robustness compared to prior approaches. Code and artifacts are available at https://github.com/lone17/angular-steering/.

Figures (14)

• Figure 1: Geometric interpretation of activation steering. Left: Before normalization, the original activation vector ${\bm{h}}$, the feature direction ${\bm{d}}_{\text{\;feat}}$, the ablation vector ${\bm{h}}_\perp$, and addition ${\bm{h}} + \alpha {\bm{d}}_{\text{\;feat}}$ are shown. Right: After normalization, all vectors lie on a (scaled) unit circle within $Span\{{\bm{h}}, {\bm{d}}_{\text{\;feat}}\}$. The dashed arc shows ${\bm{h}}_\perp$ and ${\bm{h}} + \alpha {\bm{d}}_{\text{\;feat}}$ as rotations of ${\bm{h}}$, motivating Angular Steering.
• Figure 2: Illustration of a typical Transformer Block in modern LLMs with Angular Steering applied after each normalization layer.
• Figure 3: Norms of activations at each layer of Qwen2.5-7B-Instruct for harmful and harmless samples.
• Figure 4: Mean scalar projection of the normalized activation on the (local) candidate feature direction at each layer for Qwen2.5-7B-Instruct.
• Figure 5: Statistics of refusal direction candidate for Qwen2.5-7B-Instruct.

Theorems & Definitions (3)

• Remark 1: Automatic Direction Selection
• Remark 2
• Remark 3