W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering

TL;DR

W-RAG tackles the scarcity of labeled data for dense retrievers in OpenQA by harvesting weak supervision from an LLM's ability to generate the ground-truth answer given question-passage pairs. It reranks BM25 top-k passages by the likelihood that the LLM would produce the correct answer, using the top passage as a positive example to fine-tune dense retrievers (DPR and ColBERT). Across four OpenQA datasets, W-RAG improves both retrieval and final OpenQA performance, approaching results achieved with human-labeled data and often outperforming strong baselines. The approach reduces reliance on costly annotations while delivering practical gains in RAG-based QA systems, with code and experiments designed for reproducibility.

Abstract

In knowledge-intensive tasks such as open-domain question answering (OpenQA), large language models (LLMs) often struggle to generate factual answers, relying solely on their internal (parametric) knowledge. To address this limitation, Retrieval-Augmented Generation (RAG) systems enhance LLMs by retrieving relevant information from external sources, thereby positioning the retriever as a pivotal component. Although dense retrieval demonstrates state-of-the-art performance, its training poses challenges due to the scarcity of ground-truth evidence, largely attributed to the high costs of human annotation. In this paper, we propose W-RAG, a method that draws weak training signals from the downstream task (such as OpenQA) of an LLM, and fine-tunes the retriever to prioritize passages that most benefit the task. Specifically, we rerank the top-$k$ passages retrieved via BM25 by assessing the probability that the LLM will generate the correct answer for a question given each passage. The highest-ranking passages are then used as positive fine-tuning examples for dense retrieval. We conduct comprehensive experiments across four publicly available OpenQA datasets to demonstrate that our approach enhances both retrieval and OpenQA performance compared to baseline models, achieving results comparable to models fine-tuned with human-labeled data.

Figures (5)

• Figure 1: W-RAG fits into the general RAG pipeline by training the retriever with LLM-generated weak labels. Following the steps from top to bottom, we retrieve candidate passages using BM25, present each passage to the LLM to rerank based on the answer likelihood, then use the reranked top passage to train the retriever to finally enhance LLM's response quality through the standard RAG pipeline. Prompt in "Step 2" is shown in Figure \ref{['box:weak_prompt']}.
• Figure 2: Bolded portion is evaluated for answer likelihood, which serves as a weak relevance score between "passage" and "question".
• Figure 3: Prompt for RAG OpenQA where top 5 retrieved passages are used.
• Figure 4: Same information as Table \ref{['tab: qa']}, comparing the impact of retrievers fine-tuned with weak labels versus ground-truth labels on OpenQA performance, both of which are fine-tuned from the baseline model. ReContriever serves as the baseline for the DPR-family; ColBERT initialized from base BERT serves as the baseline for the ColBERT-family.
• Figure 5: Comparison of recall for various LLMs at different top k positions, when reranking top 100 passages retrieved by BM25.