Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation

TL;DR

This work interrogates how LLMs in retrieval-augmented generation balance internal parametric knowledge with externally retrieved information. By combining macroscopic knowledge-stream analysis with microscopic module-focused methods, it identifies four streaming stages (refinement, elicitation, expression, contestation) and demonstrates that passage relevance strongly guides elicitation. The authors introduce Knowledge Activation Probability Entropy (KAPE) to locate knowledge-specific neurons and show that targeted deactivation shifts reliance between internal and external knowledge. They also reveal complementary roles for MHA and MLP in knowledge formation and verification, advancing interpretability and controllability of RAG systems for knowledge-intensive tasks. Overall, the findings offer design principles for more reliable, transparent LLM-based assistants that leverage retrieval without sacrificing factual consistency.

Abstract

Considering the inherent limitations of parametric knowledge in large language models (LLMs), retrieval-augmented generation (RAG) is widely employed to expand their knowledge scope. Since RAG has shown promise in knowledge-intensive tasks like open-domain question answering, its broader application to complex tasks and intelligent assistants has further advanced its utility. Despite this progress, the underlying knowledge utilization mechanisms of LLM-based RAG remain underexplored. In this paper, we present a systematic investigation of the intrinsic mechanisms by which LLMs integrate internal (parametric) and external (retrieved) knowledge in RAG scenarios. Specially, we employ knowledge stream analysis at the macroscopic level, and investigate the function of individual modules at the microscopic level. Drawing on knowledge streaming analyses, we decompose the knowledge utilization process into four distinct stages within LLM layers: knowledge refinement, knowledge elicitation, knowledge expression, and knowledge contestation. We further demonstrate that the relevance of passages guides the streaming of knowledge through these stages. At the module level, we introduce a new method, knowledge activation probability entropy (KAPE) for neuron identification associated with either internal or external knowledge. By selectively deactivating these neurons, we achieve targeted shifts in the LLM's reliance on one knowledge source over the other. Moreover, we discern complementary roles for multi-head attention and multi-layer perceptron layers during knowledge formation. These insights offer a foundation for improving interpretability and reliability in retrieval-augmented LLMs, paving the way for more robust and transparent generative solutions in knowledge-intensive domains.

Figures (7)

• Figure 1: Attention-based information flows across various directions in two versions of LLaMA-3 with different parameter scales, evaluated under the RAG setting using gold and fake passages on the average of three datasets.
• Figure 2: Illustrations of the four stages of knowledge streaming within RAG.
• Figure 3: Attention-based information flow based on external passages of various relevance with the query.
• Figure 4: Saliency score on three information flow directions on Natural Questions and TriviaQA dataset for LLaMA-3-8B.
• Figure 5: The figure shows the decrease in the probability of generating internal knowledge. This occurs after disrupting the key-to-query knowledge streaming across the four knowledge streaming stages.