Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Models

TL;DR

This work tackles the persistent hallucination problem in large language models by introducing END, a training-free decoding method that leverages token-wise cross-layer prediction changes to quantify the factual knowledge required for each candidate token. END computes a cross-layer distribution across intermediate layers, derives a cross-layer entropy as a factuality signal, and reweights the final token distribution to favor factually grounded tokens. Extensive experiments on TruthfulQA, FACTOR, TriviaQA, and Natural Questions demonstrate that END improves factuality and informativeness while preserving core QA performance, with evidence of generalization across model scales and backbones. The approach offers a new lens on the relationship between internal hidden states and output truthfulness at the token level, and provides a practical, training-free tool for reducing hallucinations in real-world generation tasks.

Abstract

Despite their impressive capacities, Large language models (LLMs) often struggle with the hallucination issue of generating inaccurate or fabricated content even when they possess correct knowledge. In this paper, we extend the exploration of the correlation between hidden-state prediction changes and output factuality into a deeper, token-wise level. Based on the insights , we propose cross-layer Entropy eNhanced Decoding (END), a decoding method that mitigates hallucinations without requiring extra training. END leverages inner probability changes across layers to individually quantify the factual knowledge required for each candidate token, and adjusts the final predicting distribution to prioritize tokens with higher factuality. Experiments on both hallucination and QA benchmarks demonstrate that END significantly enhances the truthfulness and informativeness of generated content while maintaining robust QA accuracy. Moreover, our work provides a deeper perspective on understanding the correlations between inherent knowledge and output factuality.

Figures (4)

• Figure 1: Illustration of our proposed method for improving LLMs' decoding factuality. The prediction probability of the wrong token 'Beijing' is adjusted and surpassed, since the correct token 'London' requires more factual knowledge during inference.
• Figure 2: (1) The left figure illustrates the predicting distribution differences, measure by KL-divergence, between the final layer and even-numbered early layers of the whole output sentence. Row names are indices of the early layer used for contrasting and column names are decoded tokens at each generation step. (2) The right figure illustrates the predicting probabilities of high probability candidate tokens among higher layers at the first generation step of token '$\_Sun$'. Row names are indices of the layer and column names are candidate tokens.
• Figure 3: Workflow of our proposed method. While the predicting probability of token A remains almost unchanged in higher layers, that of token D shows a sharp growing trend from a low value to the second-highest. The cross-layer entropy adjusts the final next-token prediction by suppressing the token A for low factuality and highlighting token D for both high prediction probability and high factuality.
• Figure 4: The distribution of prediction probability and cross-layer entropy of high probability candidates when generating the first token in response to question 'Which is the biggest airline company in the US?'