LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation

TL;DR

The paper addresses how external retrieved knowledge is integrated across transformer layers in retrieval-augmented generation (RAG) models. It reveals a three-stage layer decomposition: shallow layers handle local context, intermediate layers integrate external knowledge, and deep layers rely on internal parametric knowledge. Based on this, the authors introduce Layer Fused Decoding (LFD), a training-free decoding strategy that dynamically identifies an optimal late-stage layer using the Internal Knowledge Score (IKS) and fuses its intermediate representations with final-layer outputs to maximize grounding in retrieved evidence. Empirical results across multiple datasets and models show that LFD achieves competitive or superior accuracy to noise-based methods while incurring lower latency and memory overhead, with robust performance and minimal dataset-specific tuning. This approach provides a principled, practical means to harness external knowledge in RAG and offers insight into the internal mechanisms by which LLMs utilize retrieved information.

Abstract

Retrieval-augmented generation (RAG) incorporates external knowledge into large language models (LLMs), improving their adaptability to downstream tasks and enabling information updates. Surprisingly, recent empirical evidence demonstrates that injecting noise into retrieved relevant documents paradoxically facilitates exploitation of external knowledge and improves generation quality. Although counterintuitive and challenging to apply in practice, this phenomenon enables granular control and rigorous analysis of how LLMs integrate external knowledge. Therefore, in this paper, we intervene on noise injection and establish a layer-specific functional demarcation within the LLM: shallow layers specialize in local context modeling, intermediate layers focus on integrating long-range external factual knowledge, and deeper layers primarily rely on parametric internal knowledge. Building on this insight, we propose Layer Fused Decoding (LFD), a simple decoding strategy that directly combines representations from an intermediate layer with final-layer decoding outputs to fully exploit the external factual knowledge. To identify the optimal intermediate layer, we introduce an internal knowledge score (IKS) criterion that selects the layer with the lowest IKS value in the latter half of layers. Experimental results across multiple benchmarks demonstrate that LFD helps RAG systems more effectively surface retrieved context knowledge with minimal cost.

Figures (12)

• Figure 1: Illustration for layer-wise behavior in LLMs for RAG. Given a query and retrieved documents with the correct answer ("Real Madrid"), shallow layers capture local context, middle layers focus on answer-relevant content, while deep layers may over-rely on internal knowledge and hallucinate (e.g., "Barcelona"). Our proposal, LFD fuses middle-layer signals into the final output to preserve external knowledge and improve accuracy.
• Figure 2: Calculation procedure for SimHidden and DiffAttn metrics.
• Figure 3: (a) Average SimHidden scores (with 95% confidence intervals) across layers under varying noise levels (0, 4, 8, 12); (b) Average DiffAttn scores (with 95% confidence intervals) across layers when noise level = 0.
• Figure 4: The proposed LFD includes two key components: (1) A dynamic layer selection method using IKS to pinpoint the most impactful layers for integrating external retrieval knowledge. (2) A knowledge fusion mechanism that merges external information with the model's predictions after adaptive filtering to ensure alignment with the model's reasoning.
• Figure 5: Comparison between LFD and LFD (Fixed) on the NQ (a) and HotpotQA (b) datasets. (c) illustrates the layer selection distribution in LFD compared to the optimal fixed layer selection.