Improving Activation Steering in Language Models with Mean-Centring

TL;DR

The paper tackles the challenge of deriving effective activation steering vectors for LLMs by exploiting mean-centering: subtracting the training-activation mean from the target-activation mean yields a distillation vector that steers model outputs toward a desired behavior. This simple, inference-time technique is applied across three domains—toxicity reduction, genre steering in story continuations, and improved function-vectors—demonstrating performance gains across multiple models. By framing activations as anisotropic and biased away from the origin, mean-centring provides a robust, versatile approach that expands the applicability of activation steering beyond methods requiring explicit counterbalances. The results suggest mean-centred steering can serve as a practical tool for controllable generation in diverse NLP tasks, with future work exploring connections to activation geometry and anisotropy.

Abstract

Recent work in activation steering has demonstrated the potential to better control the outputs of Large Language Models (LLMs), but it involves finding steering vectors. This is difficult because engineers do not typically know how features are represented in these models. We seek to address this issue by applying the idea of mean-centring to steering vectors. We find that taking the average of activations associated with a target dataset, and then subtracting the mean of all training activations, results in effective steering vectors. We test this method on a variety of models on natural language tasks by steering away from generating toxic text, and steering the completion of a story towards a target genre. We also apply mean-centring to extract function vectors, more effectively triggering the execution of a range of natural language tasks by a significant margin (compared to previous baselines). This suggests that mean-centring can be used to easily improve the effectiveness of activation steering in a wide range of contexts.

Figures (9)

• Figure 1: An example of mean-centring illustrated on a set of highly anisotropic activations (i.e. offset from the origin). When steering, we want to use the vector $\mathbf{f}$ which generates some target behaviour. We compute this by averaging activations from a dataset exhibiting this behaviour, $\mu_{\text{target}}$ and subtracting the mean across all training examples $\mathbf{b}$.
• Figure 2: Results from Toxicity Removal Experiments
• Figure 3: Genre-word frequencies in generated text continuing stories of a given genre. Text generated without steering (Unsteered) is compared to text generated using mean-centring with a target genre's dataset (Mean-centred). Mean-centring consistently reduces the frequency of words in the genre we steer away from, and increases it in the genre we steer towards.
• Figure 4: Average accuracy (with 95% CI error bars) plots for steering GPT-J-6B with the uncentred and mean-centred method, as well as the average accuracy without steering.
• Figure 5: Average cosine similarity across pairs of activations in either the residual stream, the output of an Attention Layer, or the output of an MLP layer. Results are for GPT-2 small, medium, large and XL respectively. Activations were generated using the Training Subset Dataset (Appendix \ref{['app:datasets']}).