Listen to the Layers: Mitigating Hallucinations with Inter-Layer Disagreement

TL;DR

This work introduces CoCoA, a training-free inference-time decoder that mitigates LLM hallucinations by listening to inter-layer signals in the model’s middle layers. It formalizes two stability metrics, ConMLDS and fMLDS, to quantify representational disagreement across middle layers and integrates them into a decoding objective, with a self-information gated variant CoCoA-SIG to emphasize high-surprise spans. Across diverse tasks (truthful QA, summarization, code generation) and model families (Llama-3, Mistral, Qwen, CodeLlama), CoCoA and CoCoA-SIG consistently improve factuality and trustworthiness without retraining. The approach highlights the practical value of mechanistic interpretability signals for improving LLM reliability in real-world settings.

Abstract

Pretrained Large Language Models (LLMs) are prone to generating fluent yet factually incorrect text-a phenomenon known as hallucinations, undermining their reliability and utility in downstream tasks. We hypothesize that a generated text span's factuality is correlated with its representational instability across the model's internal layers. Based on this, we propose the CoCoA (Confusion and Consistency Aware) decoder, a novel, training-free decoding algorithm that mitigates hallucinations at inference time by listening to these signals in the middle layers. We propose two metrics to quantify this instability in the middle layers, and use it to penalize outputs that exhibit high internal confusion, thereby steering the model towards more internally consistent and factually grounded outputs. We further propose a self-information gated variant, CoCoA-SIG, that dynamically modulates this penalty to selectively target high-surprise, unstable generations. Extensive experiments on diverse tasks, including question-answering, summarization and code generation demonstrate that CoCoA significantly improves factual correctness across multiple model families (e.g., Llama-3, Qwen-2.5, Mistral). By leveraging model-intrinsic signals, CoCoA offers an effective and broadly applicable method for enhancing the trustworthiness of LLMs at inference time, without requiring any model retraining.

Figures (7)

• Figure 1: Comparison between Greedy decoding and CoCoA decoding for Llama-3-8b-Instruct. CoCoA decoding improves LLM's reliability by generating truthful responses.
• Figure 2: Overview of the proposed CoCoA decoding framework. At each decoding step (1) The LLM generates multiple candidate spans. (2) For each candidate span, we extract hidden state representations from selected middle layers. (3) For each span, we compute the Middle Layer Disagreement Score (MLDS), which quantifies representational inconsistency across layers as proposed in Eqns \ref{['eqn:conMLDS-def']} and \ref{['eqn:fMLDS-def']} (4) The CoCoA decoder combines forward log-probability with MLDS to produce a unified score, and finally, the span with the highest combined score—corresponding to low middle-layer confusion and high consistency—is selected as the output.
• Figure 3: Visualization of the effect of the CoCoA and ${\rm CoCoA}_{\rm SIG}$ decoder on span level decisions. The x-axis represents different candidate spans, and y-axis shows their fMLDS score. Our decoder moves the decision towards the non-hallucinated output as shown by the arrows.
• Figure 4: Prompt template for evaluating truthfulness of model responses for TruthfulQA generation task.
• Figure 5: Prompt template for evaluating informativeness of model responses on TruthfulQA generation task.