AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint
Leheng Sheng, Changshuo Shen, Weixiang Zhao, Junfeng Fang, Xiaohao Liu, Zhenkai Liang, Xiang Wang, An Zhang, Tat-Seng Chua
TL;DR
AlphaSteer addresses the safety–utility trade-off in activation steering for refusing malicious prompts by introducing a learnable, null-space-constrained framework. It learns layer-wise transformations $\mathbf{\Delta}^{(l)}$ to steer activations via $\mathbf{h}^{(l)'} = \mathbf{h}^{(l)} + \lambda \mathbf{\Delta}^{(l)} \mathbf{h}^{(l)}$, while enforcing $\mathbf{\Delta} \mathbf{H}_b = \mathbf{0}$ through a null-space projection $\hat{\mathbf{P}}$ so benign prompts remain unchanged. For malicious prompts, AlphaSteer reconstructs a refusal direction with a regularized least-squares objective, yielding a closed-form solution $\tilde{\mathbf{\Delta}}^{\star}$ that, when applied, biases activations toward refusal, thereby enhancing safety. Across multiple LLMs and jailbreak attacks, AlphaSteer achieves high defense success rates with minimal utility loss, and experiments reveal favorable activation-space dynamics and robustness to unseen threats, all while retaining practical inference efficiency and providing open-source implementation.
Abstract
As LLMs are increasingly deployed in real-world applications, ensuring their ability to refuse malicious prompts, especially jailbreak attacks, is essential for safe and reliable use. Recently, activation steering has emerged as an effective approach for enhancing LLM safety by adding a refusal direction vector to internal activations of LLMs during inference, which will further induce the refusal behaviors of LLMs. However, indiscriminately applying activation steering fundamentally suffers from the trade-off between safety and utility, since the same steering vector can also lead to over-refusal and degraded performance on benign prompts. Although prior efforts, such as vector calibration and conditional steering, have attempted to mitigate this trade-off, their lack of theoretical grounding limits their robustness and effectiveness. To better address the trade-off between safety and utility, we present a theoretically grounded and empirically effective activation steering method called AlphaSteer. Specifically, it considers activation steering as a learnable process with two principled learning objectives: utility preservation and safety enhancement. For utility preservation, it learns to construct a nearly zero vector for steering benign data, with the null-space constraints. For safety enhancement, it learns to construct a refusal direction vector for steering malicious data, with the help of linear regression. Experiments across multiple jailbreak attacks and utility benchmarks demonstrate the effectiveness of AlphaSteer, which significantly improves the safety of LLMs without compromising general capabilities. Our codes are available at https://github.com/AlphaLab-USTC/AlphaSteer.