Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Models

TL;DR

This work identifies reliability gaps in inference-time steering using static steering vectors and introduces Steering Vector Fields (SVF), which model context-dependent steering as a vector field derived from a differentiable concept boundary. By mapping representations from multiple layers into a shared space and using the boundary's local normal as the steering direction, SVF enables coordinated multi-layer interventions and robust control during long-form generation and multi-attribute steering. The approach is validated across multiple LLMs and steering tasks, showing stronger, more reliable control than baselines while preserving model utility and generalization, including under distribution shifts. Overall, SVF advances practical inference-time control by aligning concept geometry with current activations and across depth, supporting dynamic, context-aware steering without parameter updates.

Abstract

Steering vectors (SVs) offer a lightweight way to control large language models (LLMs) at inference time by shifting hidden activations, providing a practical middle ground between prompting and fine-tuning. Yet SVs can be unreliable in practice. Some concepts are unsteerable, and even when steering helps on average it can backfire for a non-trivial fraction of inputs. Reliability also degrades in long-form generation and multi-attribute steering. We take a geometric view of these failures. A static SV applies the same update vector everywhere in representation space, implicitly assuming that the concept-improving direction is constant across contexts. When the locally effective direction varies with the current activation, a single global vector can become misaligned, which yields weak or reversed effects. Guided by this perspective, we propose Steering Vector Fields (SVF), which learns a differentiable concept scoring function whose local gradient defines the steering direction at each activation, making interventions explicitly context-dependent. This formulation supports coordinated multi-layer interventions in a shared, aligned concept space, and enables efficient long-form and multi-attribute control within a unified framework. Across multiple LLMs and steering tasks, SVF delivers stronger and more reliable control, improving the practicality of inference-time steering.

Figures (14)

• Figure 1: A global steering vector can become misaligned with the locally effective direction, leading to unsteerable or anti-steerable behavior; context-dependent steering mitigates this failure mode.
• Figure 2: CAA versus KNN steering. KNN provides a simple context-conditioned rule that adapts the steering direction to the current state, enabling an empirical check of context dependence.
• Figure 3: Test accuracy versus displacement budget (the injected shift magnitude $\|\alpha v\|$ in Eq. \ref{['eq:steer_formulation']}) for CAA and KNN steering.
• Figure 4: Pipeline of Steering Vector Fields (SVF). SVF extracts hidden states from a set of layers, projects and calibrates them into a shared concept space, learns a single cross-layer boundary, and then uses the boundary normal to produce coordinated per-layer steering.
• Figure 5: Kernel density estimates of per-example steerability comparing SVF and CAA. Steerability is the fitted slope of the logit-gap curve as $\alpha$ is swept over each method’s effective interval (positive indicates steering; negative indicates anti-steering).