ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models

TL;DR

ContextFocus introduces activation steering to improve contextual faithfulness in LLMs facing knowledge conflicts, without requiring finetuning and with minimal inference overhead. By constructing a contrastive steering vector $v^{(l)}$ from context-present versus context-absent prompts and injecting it at an intermediate layer with multiplier $m$, the method biases generation toward context-grounded outputs while preserving fluency. Empirical results on ConFiQA across multiple model families show substantial gains in context-faithful outputs (higher $p_s$, lower $p_o$ and $M_R$), data-efficient vector estimation (roughly 1.5k examples), and compatibility with prompting strategies, all with single-pass inference. These findings establish activation-level control as a practical, scalable tool for robust, context-grounded generation in retrieval-augmented settings.

Abstract

Large Language Models (LLMs) encode vast amounts of parametric knowledge during pre-training. As world knowledge evolves, effective deployment increasingly depends on their ability to faithfully follow externally retrieved context. When such evidence conflicts with the model's internal knowledge, LLMs often default to memorized facts, producing unfaithful outputs. In this work, we introduce ContextFocus, a lightweight activation steering approach that improves context faithfulness in such knowledge-conflict settings while preserving fluency and efficiency. Unlike prior approaches, our solution requires no model finetuning and incurs minimal inference-time overhead, making it highly efficient. We evaluate ContextFocus on the ConFiQA benchmark, comparing it against strong baselines including ContextDPO, COIECD, and prompting-based methods. Furthermore, we show that our method is complementary to prompting strategies and remains effective on larger models. Extensive experiments show that ContextFocus significantly improves contextual-faithfulness. Our results highlight the effectiveness, robustness, and efficiency of ContextFocus in improving contextual-faithfulness of LLM outputs.

Figures (5)

• Figure 1: When the model's memory deviates from the context, the model may generate outputs that are unfaithful to the context. In the example shown here, the LLM was likely trained before the CEO of Starbucks changed to Brian Niccol. By applying our steering method with a multiplier of 2, we see that the model now generates a faithful output.
• Figure 2: Layer-wise evaluation of steering on Llama-3.1-8B. We apply the +2 multiplier to steering vectors across layers for 200 held-out open-ended questions from the NQ-SWAP dataset. Performance is measured using the $p_s$ metric. The green dotted line denotes the unsteered ($m{=}0$) baseline. The 13th layer shows the best performance and is selected as the optimal layer.
• Figure 3: The best layer is chosen by looking at the layer with the biggest deviation in context-focus accuracy.
• Figure 4: Layer-wise evaluation of steering on Llama-3.1-70B-Instruct. We apply the +2 multiplier to steering vectors across layers for 200 held-out open-ended questions from the NQ-SWAP dataset. Performance is measured using the $p_s$ metric. The green dotted line denotes the unsteered ($m{=}0$) baseline. The 32nd layer shows the best performance and is selected as the optimal layer.
• Figure 5: Layer-wise evaluation of steering on Mistral-7B-Instructv0.3. We apply the +2 multiplier to steering vectors across layers for 200 held-out open-ended questions from the NQ-SWAP dataset. Performance is measured using the $p_s$ metric. The green dotted line denotes the unsteered ($m{=}0$) baseline. The 11th layer shows the best performance and is selected as the optimal layer.