GRAD: Graph-Retrieved Adaptive Decoding for Hallucination Mitigation

TL;DR

Graph-Retrieved Adaptive Decoding (GRAD), a decoding-time method that grounds generation in corpus-derived evidence without retraining, is introduced, demonstrating that statistical evidence from corpus-level token transitions can effectively steer generation toward more truthful and verifiable outputs.

Abstract

Hallucination mitigation remains a persistent challenge for large language models (LLMs), even as model scales grow. Existing approaches often rely on external knowledge sources, such as structured databases or knowledge graphs, accessed through prompting or retrieval. However, prompt-based grounding is fragile and domain-sensitive, while symbolic knowledge integration incurs heavy retrieval and formatting costs. Motivated by knowledge graphs, we introduce Graph-Retrieved Adaptive Decoding (GRAD), a decoding-time method that grounds generation in corpus-derived evidence without retraining. GRAD constructs a sparse token transition graph by accumulating next-token logits across a small retrieved corpus in a single forward pass. During decoding, graph-retrieved logits are max-normalized and adaptively fused with model logits to favor high-evidence continuations while preserving fluency. Across three models and a range of question-answering benchmarks spanning intrinsic, extrinsic hallucination, and factuality tasks, GRAD consistently surpasses baselines, achieving up to 9.7$\%$ higher intrinsic accuracy, 8.6$\%$ lower hallucination rates, and 6.9$\%$ greater correctness compared to greedy decoding, while attaining the highest truth--informativeness product score among all methods. GRAD offers a lightweight, plug-and-play alternative to contrastive decoding and knowledge graph augmentation, demonstrating that statistical evidence from corpus-level token transitions can effectively steer generation toward more truthful and verifiable outputs.

Figures (5)

• Figure 1: Overview of Graph-Retrieved Adaptive Decoding (GRAD). Example from WikiQA dataset yang2015wikiqa. The greedy decoding output ("The last La Nina event occurred from October 2020") is incorrect, whereas GRAD correctly predicts "July 2020". GRAD improves faithfulness by refining next-token predictions using logit signals retrieved from a precomputed Token Transition Graph (TTG). The TTG is built from a small corpus by accumulating next-token logits across overlapping contexts in a single forward pass. During decoding, retrieved logits are max-normalized and adaptively fused with model logits, promoting high-evidence continuations while preserving fluency. While the exact context for predicting the next token after "from" is not represented in the TTG, the graph still successfully guides decoding toward the space token "_" (used illustratively to denote a space) and avoids uncertain continuations where multiple candidates (e.g., "_May", "_October", "_") have comparable logit values. Additional qualitative case studies are provided in Appendix \ref{['app:case-study']}.
• Figure 2: Effect of training corpus size $|\mathcal{D}|$ on GRAD performance (Qwen2.5-3B). Detailed numbers in Table \ref{['tab:ablation_corpus_detailed']} (Appendix \ref{['app:detailed-results']}).
• Figure 3: Graph growth vs. $|\mathcal{D}|$: nodes (solid) and edges (dashed). Detailed results in Table \ref{['tab:graph-stats']} (Appendix \ref{['app:detailed-results']}.)
• Figure 4: Effect of $\alpha$ across three benchmarks using Qwen2.5-3B. $\alpha=0$ corresponds to greedy decoding. Detailed results in Table \ref{['tab:ablation_alpha_detailed']} (Appendix \ref{['app:detailed-results']}).
• Figure 5: Informativeness evaluation prompt used for WikiQA.