Extending Activation Steering to Broad Skills and Multiple Behaviours

TL;DR

This work extends activation steering from narrow skills to broad skills and multiple behaviours in large language models, evaluating two paradigms: broad steering and multi-steering. Using LLama 2 7b Chat, it demonstrates that broad skills like coding can be steered with comparable effectiveness to narrower skills and that steering multiple behaviours individually is feasible, though combining them into a single steering vector is generally less effective. Simultaneous steering across multiple layers shows promise in steering several behaviours with modest alignment tax, offering a practical path to more nuanced risk mitigation. The results illuminate the trade-offs and design considerations for applying activation steering to broader latent spaces and multiple behaviours.

Abstract

Current large language models have dangerous capabilities, which are likely to become more problematic in the future. Activation steering techniques can be used to reduce risks from these capabilities. In this paper, we investigate the efficacy of activation steering for broad skills and multiple behaviours. First, by comparing the effects of reducing performance on general coding ability and Python-specific ability, we find that steering broader skills is competitive to steering narrower skills. Second, we steer models to become more or less myopic and wealth-seeking, among other behaviours. In our experiments, combining steering vectors for multiple different behaviours into one steering vector is largely unsuccessful. On the other hand, injecting individual steering vectors at different places in a model simultaneously is promising.

Figures (10)

• Figure 1: Sample template.
• Figure 3: This figure illustrates the effect of applying a steering vector aiming to remove coding ability. The scores for text and code data are recorded. Each line represents steering at a certain layer, and each dot represents the scores for one injection coefficient. The horizontal dotted line illustrates perfect performance (which is impossible for our data due to noise), and the diagonal dotted line shows equal performance drops in code and text.
• Figure 4: This figure illustrates the effect of applying a steering vector aiming to remove Python ability, in the same way as in Figure \ref{['fig:general label for coding']}.
• Figure 5: The results for individual steering for layer 15. The used injection coefficients can be found in Table \ref{['tab:injection coefficients single steering']}
• Figure 6: Combining the individual steering vectors into one injected in layer 15. The combinations differ in three dimensions: take the mean or sum, weighted or unweighted, and subtracted or added. We compare the combined steering to the individual steering presented in Figure \ref{['fig: single steering layer 15']}, which are indicated with the grey horizontal lines.