Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering

TL;DR

SpotLight presents an inference-time mechanism to amplify user-specified prompt spans by dynamically steering Transformer attention toward them, without weight updates. The method computes the current attention share on the spans, and if below a target $\psi_{\mathrm{target}}$, adds a log-space bias to the span logits, applied across all layers and heads. Across multiple datasets (IFEval, ManyIFEval, MT-IFEval, ComplexBench, WildJailbreak, CoCoNot) and model scales (3B–72B), SpotLight yields substantial gains in instruction-following accuracy while preserving core task performance, and exhibits robustness to longer contexts and diverse instruction types. Compared to fixed-attention baselines like PASTA, it requires no offline profiling and introduces only modest inference-time overhead, making it practical for real-world deployment. The results suggest dynamic attention steering as a flexible tool to align model behavior with user intent in complex, evolving instruction scenarios.

Abstract

In many real-world applications, users rely on natural language instructions to guide large language models (LLMs) across a wide range of tasks. These instructions are often complex, diverse, and subject to frequent change. However, LLMs do not always attend to these instructions reliably, and users lack simple mechanisms to emphasize their importance beyond modifying prompt wording or structure. To address this, we present an inference-time method that enables users to emphasize specific parts of their prompt by steering the model's attention toward them, aligning the model's perceived importance of different prompt tokens with user intent. Unlike prior approaches that are limited to static instructions, require significant offline profiling, or rely on fixed biases, we dynamically update the proportion of model attention given to the user-specified parts--ensuring improved instruction following without performance degradation. We demonstrate that our approach improves instruction following across a variety of tasks involving multiple instructions and generalizes across models of varying scales.

Figures (9)

• Figure 1: Visualizing token attentions for different predictions with the Qwen2.5 7B Instruct model.
• Figure 2: Multi-turn instruction following with MT-IFEval. As the context increases with the number of turns, the performance of all methods drop, with SpotLight achieving better performance across turns and remaining the most robust with the least drop across all models.
• Figure 3: Performance comparison on the ComplexBench dataset. SpotLight improves model performance on most constraint categories.
• Figure 4: Performance comparison on the WildJailbreak dataset. SpotLight improves general and exact refusal capabilities across all models and maintains comparable performance on benign queries.
• Figure 5: Impact of instruction placement -- SpotLight outperforms prompting strategies like repeating instructions or placing them before / after the user query. SpotLight can also be used in conjunction with these strategies to further improve performance.