REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering

TL;DR

This work develops a novel architecture for LLM based RAG system, by incorporating a specially designed assessnent module that precisely assesses the relevance of retrieved documents, and proposes an improved training method based on bi-granularity relevance fusion and noise-resistant training.

Abstract

Considering the limited internal parametric knowledge, retrieval-augmented generation (RAG) has been widely used to extend the knowledge scope of large language models (LLMs). Despite the extensive efforts on RAG research, in existing methods, LLMs cannot precisely assess the relevance of retrieved documents, thus likely leading to misleading or even incorrect utilization of external knowledge (eg., retrieved documents). To address this issue, in this paper, we propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA). As the key motivation, we aim to enhance the self-awareness regarding the reliability of external knowledge for LLMs, so as to adaptively utilize external knowledge in RAG systems. Specially, we develop a novel architecture for LLM-based RAG systems, by incorporating a specially designed assessment module that precisely assesses the relevance of retrieved documents. Furthermore, we propose an improved training method based on bi-granularity relevance fusion and noise-resistant training. By combining the improvements in both architecture and training, our proposed REAR can better utilize external knowledge by effectively perceiving the relevance of retrieved documents. Experiments on four open-domain QA tasks show that REAR significantly outperforms previous a number of competitive RAG approaches. Our codes can be accessed at https://github.com/RUCAIBox/REAR.

Figures (6)

• Figure 1: LLMs may be misled by irrelevant documents, and struggle to determine the relevance of a document ren2023investigatingzhang2024large.
• Figure 2: The overview of the proposed REAR framework.
• Figure 3: Results of RAG performance vary in overall document count and quality. The left one presents RAG performance with varying numbers of retrieved documents. The right one is the results of RAG with different retriever engines. R1, R2, and R3 represent BM25, Contriever-msmarco, and the FiD-distilled retriever, R1 < R2 < R3 (Table \ref{['tab:nq-re']} of the Appendix) .
• Figure 4: The illustration of different retrieved documents and different labeling metrics.
• Figure 5: Prompts for "direct RAG QA".