Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models

TL;DR

This work tackles context-faithfulness hallucinations in retrieval-augmented LLMs by showing that token-level uncertainty strongly correlates with unfaithful outputs and that retrieved context is often recognized but not properly prioritized. It introduces Dynamic Attention-Guided Context Decoding (DAGCD), a single-pass decoding framework that leverages attention-utilization signals via a Context Utilization Detector and a top-K attention-head analysis to construct a utilization distribution, then adjusts token probabilities using $P' = P + \alpha H_{ ext{norm}}(P) \cdot U_{top}$ to bias generation toward contextually supported tokens. The approach is validated across seven open-book QA datasets and multiple model families, yielding substantial improvements in EM and F1 over greedy decoding and competitive baselines, with strong data-efficiency and real-time applicability. The work also shows that attention-head signals offer a robust, interpretable mechanism for understanding and extracting context-utilization patterns, suggesting a fundamental role for attention in controlling context integration in Transformer-based LLMs.

Abstract

Large language models (LLMs) often exhibit Context Faithfulness Hallucinations, where outputs deviate from retrieved information due to incomplete context integration. Our analysis reveals a strong correlation between token-level uncertainty and hallucinations. We hypothesize that attention mechanisms inherently encode context utilization signals, supported by probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD), a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. Experiments on open-book QA datasets demonstrate DAGCD's effectiveness, yielding significant improvements in faithfulness and robustness while preserving computational efficiency.

Figures (15)

• Figure 1: Analysis of the token-level probability distribution after context concatenation: (a1, a2) the model's uncertainty when generating correct versus wrong answers, measured by NE and MSP; (b) for wrong answers, the ranking of the golden answer token within the token-level probability distribution.
• Figure 2: Cross Domain Validation. One dataset as the training set (X-axis) and the remaining datasets as the test sets, showing the AUC on the test sets (Y-axis).
• Figure 3: Training Data Size Validation. Training sets of varying sizes were constructed from a single dataset (HotpotQA), and evaluate on remain datasets.
• Figure 4: The illustration of the generation process of our proposed DAGCD method.
• Figure 5: Ablation Study for DAGCD. Left Part: Ablation 1 Detector Training Data Sizes, Center Part: Ablation 2 Top-Rank Constraint, Right Part: Ablation 3 Scaling Factor $\alpha$.