Understanding (Un)Reliability of Steering Vectors in Language Models

TL;DR

This work probes the reliability of steering vectors for controlling LLM behavior via inference-time activation biases. Using Contrastive Activation Addition across 36 Anthropic evaluation datasets with a Llama2-7B-Chat model, it shows that all seven prompt types yield a net-positive steering effect on average but with high variability and non-negligible anti-steerability, and that steering vectors can diverge directionally across prompts. The authors demonstrate that datasets with higher directional agreement between training activation differences and the steering vector, as well as better separability of positive and negative activations along the difference-of-means line, exhibit stronger steerability. These findings provide diagnostic criteria for when vector steering is likely to be reliable and suggest that steering methods should account for activation-space geometry to achieve robust, interpretable control of language models.

Abstract

Steering vectors are a lightweight method to control language model behavior by adding a learned bias to the activations at inference time. Although steering demonstrates promising performance, recent work shows that it can be unreliable or even counterproductive in some cases. This paper studies the influence of prompt types and the geometry of activation differences on steering reliability. First, we find that all seven prompt types used in our experiments produce a net positive steering effect, but exhibit high variance across samples, and often give an effect opposite of the desired one. No prompt type clearly outperforms the others, and yet the steering vectors resulting from the different prompt types often differ directionally (as measured by cosine similarity). Second, we show that higher cosine similarity between training set activation differences predicts more effective steering. Finally, we observe that datasets where positive and negative activations are better separated are more steerable. Our results suggest that vector steering is unreliable when the target behavior is not represented by a coherent direction.

Figures (10)

• Figure 1: Steering vectors trained with different prompt types all increase the mean logit-difference relative to no steering and perform similarly across datasets. Yet, for all prompt types, steering effect size is unreliable, with a significant fraction of the test samples shifted in the opposite direction ("anti-steerable"). Both steering effect size and faction of such anti-steerable samples vary substantially between datasets, as shown by the six most steerable datasets (top row) outperforming the average shown (bottom row) in both metrics. We used 250 training samples and 500 evaluation samples for each combination of prompt type and dataset.
• Figure 2: We group the 36 datasets by how effective the resulting steering vector is ("steerability rank"). The most steerable group (ranks 1-6) exhibit high directional agreement between the individual activation differences and the steering vectors, whereas directions in the least steerable group (ranks 31-36) are more dispersed or even orthogonal. Conceptually, high directional agreement suggests a coherent linear representation of the behavior.
• Figure 3: For datasets where the behavior is steerable, activations are clearly separated along the difference-of-activation-means line (top). Less steerable datasets have overlapping positive and negative activations (bottom). CAA steering shifts activations along the difference-of-means line.
• Figure 4: Steering vectors (SVs) trained on the same datasets but with different prompt types have cosine similarities ranging from 0.07 to 0.86. SVs trained with similar prompt types have higher cosine similarity than for different prompt types. Cosine similarities between SVs from prefilled prompts range from 0.25 to 0.86. Cosine similarities between SVs from non-prefilled prompts range from 0.32 and 0.44. One straightforward reason for why prefilled and non-prefilled activation differences are not similar is because generating an answer token (A/B, Yes/No) requires different computations/representations than generating the token after the answer token. Very similar prompts (prefilled 5-shot, prefilled instruction and prefilled instruction 5-shot) have comparatively high cosine similarities (0.61 to 0.86). The ranking counts for prompt types show that now single prompt type is systematically better than the others, if compared by their dataset wise mean logit-difference.
• Figure 5: Steering vectors trained with different prompt types all increase the mean logit-difference relative to no steering and perform similarly across datasets. Yet, for all prompt types, steering effect size is unreliable, with 29% - 43% of all samples shifted in the opposite direction. Both steering effect size and faction of such anti-steerable samples vary substantially between datasets, as shown by the six most steerable datasets (top row) outperforming those in the middle row (average) and the bottom row (six least steerable datasets).For the six least steerable datasets the mean logit difference compared to no steering is negative for some prompt types and the fraction of anti-steerable samples around half. We used 250 training samples and 500 evaluation samples for each combination of prompt type and dataset