Refiner: Restructure Retrieval Content Efficiently to Advance Question-Answering Capabilities

TL;DR

This work tackles hallucinations in knowledge-intensive QA by addressing the lost-in-the-middle problem in RAG. It introduces Refiner, an end-to-end extract-and-restructure paradigm that verbatim-extracts query-relevant content and organizes it into sections to reveal information interrelations for downstream LMs. Refiner is trained via multi-teacher knowledge distillation and fine-tuned as a smaller model to balance performance and computation, yielding strong gains on multi-hop QA and competitive results on single-hop QA with high token compression. Empirical results across open-domain tasks demonstrate Refiner's robustness to noisy content, compatibility with various RAG setups, and clear practical value as a plug-and-play enhancement for retrieval-augmented systems.

Abstract

Large Language Models (LLMs) are limited by their parametric knowledge, leading to hallucinations in knowledge-extensive tasks. To address this, Retrieval-Augmented Generation (RAG) incorporates external document chunks to expand LLM knowledge. Furthermore, compressing information from document chunks through extraction or summarization can improve LLM performance. Nonetheless, LLMs still struggle to notice and utilize scattered key information, a problem known as the "lost-in-the-middle" syndrome. Therefore, we typically need to restructure the content for LLM to recognize the key information. We propose $\textit{Refiner}$, an end-to-end extract-and-restructure paradigm that operates in the post-retrieval process of RAG. $\textit{Refiner}$ leverages a single decoder-only LLM to adaptively extract query-relevant contents verbatim along with the necessary context, and section them based on their interconnectedness, thereby highlights information distinction, and aligns downstream LLMs with the original context effectively. Experiments show that a trained $\textit{Refiner}$ (with 7B parameters) exhibits significant gain to downstream LLM in improving answer accuracy, and outperforms other state-of-the-art advanced RAG and concurrent compressing approaches in various single-hop and multi-hop QA tasks. Notably, $\textit{Refiner}$ achieves a 80.5% tokens reduction and a 1.6-7.0% improvement margin in multi-hop tasks compared to the next best solution. $\textit{Refiner}$ is a plug-and-play solution that can be seamlessly integrated with RAG systems, facilitating its application across diverse open-source frameworks.

Figures (5)

• Figure 1: The example shows that misleading document chunks are prone to impeding downstream LM from comprehension and leading to hallucination. While our Refiner mitigates the problem by extracting and structuring query-relevant contents by sectioning distinct information, making downstream LM disseminate them effortlessly.
• Figure 2: Knowledge distillation for Refiner. The curated training data is generated by multiple teacher models, which undergoes parsing, filtering, voting to improve section and context expediency.
• Figure 3: The ratio of contents extracted by Refiner that are exactly originate from respective document chunks under various top retrieval settings and tasks.
• Figure 4: Prepare curated train dataset for Refiner.
• Figure 5: The structure of output generated by Refiner consists of three parts: section, Title and content, whereSection elucidates relatedness information among extracted contents, Title represents the Title of the Original document, content represents the question-relevant content extracted from that document chunk.