Activation Scaling for Steering and Interpreting Language Models

TL;DR

It is argued that successfully intervening on a model is a prerequisite for interpreting its internal workings, and a three-term objective is established: a successful intervention should flip the correct with the wrong token and vice versa (effectiveness), and leave other tokens unaffected (faithfulness), all while being sparse (minimality).

Abstract

Given the prompt "Rome is in", can we steer a language model to flip its prediction of an incorrect token "France" to a correct token "Italy" by only multiplying a few relevant activation vectors with scalars? We argue that successfully intervening on a model is a prerequisite for interpreting its internal workings. Concretely, we establish a three-term objective: a successful intervention should flip the correct with the wrong token and vice versa (effectiveness), and leave other tokens unaffected (faithfulness), all while being sparse (minimality). Using gradient-based optimization, this objective lets us learn (and later evaluate) a specific kind of efficient and interpretable intervention: activation scaling only modifies the signed magnitude of activation vectors to strengthen, weaken, or reverse the steering directions already encoded in the model. On synthetic tasks, this intervention performs comparably with steering vectors in terms of effectiveness and faithfulness, but is much more minimal allowing us to pinpoint interpretable model components. We evaluate activation scaling from different angles, compare performance on different datasets, and make activation scalars a learnable function of the activation vectors themselves to generalize to varying-length prompts.

Figures (9)

• Figure 1: We show that it is often sufficient to scale a few influential activation vectors ${\color{MacroColor} \mathbf{h}_{{\color{MacroColor} l}, {\color{MacroColor} i}}^{{({\color{MacroColor} s})}}}$ for a model to favor one answer token over another token. This could be the MLP output at layer ${\color{MacroColor} l}$ for token position ${\color{MacroColor} i}$. We learn multiplicative scalars, ${\color{MacroColor} \alpha_{{\color{MacroColor} l},{\color{MacroColor} i}}^{({\color{MacroColor} s})}}$, using gradient-based optimization. These correspond to interpretable interventions that generalize to test set prompts while relying on fewer parameters than additive steering vectors.
• Figure 2: Pareto fronts that visualize the trade-off between effectiveness and faithfulness (left) and effectiveness and minimality (right) on train (crosses) and test sets (points). We compare ActivScalar and SteerVec for different hyperparameter combinations of ${\color{MacroColor} \lambda_{\mathsmaller{\textrm{F}}}}, {\color{MacroColor} \lambda_{\mathsmaller{\textrm{M}}}}, {\color{MacroColor} m} \in \{0, 1, 10, 100\}$. We learn interventions for the sites attnOut and mlpOut on all layers and token positions of GPT2-Small. We find that ActivScalar does not fall behind SteerVec in terms of effectiveness and faithfulness, but is much more minimal on average.
• Figure 3: The magnitude and sign of learned activation scalars highlight task-relevant locations within the model, here for the residPost site on GPT2-XL. For instance, the correct answer token Tom is promoted around layers 15.0 to 20.0 while activation vectors at the token Anne are scaled down. The intervention is successful according to our effectiveness objective illustrated by the red and blue lines crossing between $\beta={-}1$ and $\beta=1$, which reverses the sign of the logit difference.
• Figure 4: We interpret the learned steering vectors by comparing their difference in norm and direction, before and after training, here for the residPost site of GPT2-XL. We observe that the change in norm highlights similar token positions as ActivScalar in \ref{['fig:interpret_scalars']}. When fitting the vectors multiple times, the vectors converge to different directions every time, while changes in norm are more constant. We quantify this using the Kendall correlation $\rho$ between the orderings of locations by the change in norm as well as cosine distance between different runs.
• Figure 5: DynScalar generalizes to prompts of varying length and yields interpretable activation scalars. Here for residPost of GPT2-Small, the correct answer token is identified despite being in a different position.