Improving Instruction-Following in Language Models through Activation Steering

TL;DR

This work introduces activation steering, a mechanism that derives instruction-specific activation directions from input pairs to steer language models at inference time. By adding a residual-stream vector, the method enforces constraints on format, length, and word usage, and it generalizes across multiple models, including cross-model transfers from instruction-tuned to base models. The approach enables simultaneous multi-constraint steering and demonstrates tangible gains in instruction adherence, even without explicit instructions in the prompt, while maintaining reasonable output quality. The authors provide extensive experiments on the IFEval dataset, analyze representations, and release code and data to support reproducibility and practical adoption.

Abstract

The ability to follow instructions is crucial for numerous real-world applications of language models. In pursuit of deeper insights and more powerful capabilities, we derive instruction-specific vector representations from language models and use them to steer models accordingly. These vectors are computed as the difference in activations between inputs with and without instructions, enabling a modular approach to activation steering. We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion, providing inference-time control over instruction following. Our experiments across four models demonstrate how we can use the activation vectors to guide models to follow constraints even without explicit instructions and to enhance performance when instructions are present. Additionally, we explore the compositionality of activation steering, successfully applying multiple instructions simultaneously. Finally, we demonstrate that steering vectors computed on instruction-tuned models can transfer to improve base models. Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation. Our code and data are available at https://github.com/microsoft/llm-steer-instruct.

Figures (20)

• Figure 1: Instruction Steering Process. Steering vectors are computed as the difference in residual stream activations between inputs with and without the instruction. These vectors are then applied during inference to adjust the model’s activations, guiding it to follow the desired instruction.
• Figure 2: Residual stream similarity across layers. Phi-3's residual stream activations show higher cosine similarity between examples with the same instruction compared to those without, indicating effective capture of instruction-relevant features.
• Figure 3: Format Instructions. (a) Instruction-following accuracy without explicit text instructions shows significant improvement with steering across all models. (b) Steering enhances accuracy even when text instructions are provided. (c) Per-instruction accuracy for Phi-3 without text instructions.
• Figure 4: Quality Score. Average response quality score changes for four models, comparing the effects of steering with and without input instructions to the effect of adding instructions without steering. Steering decreases quality similarly to adding instructions.
• Figure 5: Length Instructions. (a) Modulating the steering weight $c$ effectively adjusts response length, with larger values leading to more concise outputs. (b) Steering enhances adherence to the maximum length constraints when they are explicitly specified in the input text. (c) Shift in the response length distribution upon steering for outputs constrained to a maximum of 5 sentences.