Re3val: Reinforced and Reranked Generative Retrieval

TL;DR

Re3val presents a context-aware generative retrieval framework that augments page-title generation with Dense Passage Retrieval context and reinforcement learning. By combining distant supervision via question generation, REINFORCE-based optimization, and context-driven reranking at both the page-title and context levels, it achieves state-of-the-art Top-1 KILT scores across five datasets with relatively limited pretraining data. The architecture includes a page-title reranker and a context reranker that ground answers on reranked contexts, followed by a FiD reader. Empirical results show consistent gains over baselines in zero-shot and, to a mixed extent, few-shot settings, and demonstrate data-efficient improvements in R-Precision and KILT scores, highlighting practical benefits for knowledge-intensive NLP tasks.

Abstract

Generative retrieval models encode pointers to information in a corpus as an index within the model's parameters. These models serve as part of a larger pipeline, where retrieved information conditions generation for knowledge-intensive NLP tasks. However, we identify two limitations: the generative retrieval does not account for contextual information. Secondly, the retrieval can't be tuned for the downstream readers as decoding the page title is a non-differentiable operation. This paper introduces Re3val, trained with generative reranking and reinforcement learning using limited data. Re3val leverages context acquired via Dense Passage Retrieval to rerank the retrieved page titles and utilizes REINFORCE to maximize rewards generated by constrained decoding. Additionally, we generate questions from our pre-training dataset to mitigate epistemic uncertainty and bridge the domain gap between the pre-training and fine-tuning datasets. Subsequently, we extract and rerank contexts from the KILT database using the rerank page titles. Upon grounding the top five reranked contexts, Re3val demonstrates the Top 1 KILT scores compared to all other generative retrieval models across five KILT datasets.

Figures (4)

• Figure 1: Re3val's Page Title Reranker ($g_\phi$) enhances generated page titles ($X$) with DPR contextual information ($Y$), producing reranked titles ($Z$). This is crucial when documents in $X$ lack a suitable answer to a query ($q$), as depicted in the figure.
• Figure 2: Re3val Training Pipeline. Generated questions after filtering are integrated into pre-training (1), followed by few-shot training (3) with REINFORCE (2, 4). Retrieved DPR contexts (5), perturbed page titles (6), and queries are concatenated for reranker training (7). Gold and negative passages retrieved with BM-25 are employed (8) for context reranker training (9). Contexts are retrieved using the top 5 reranked titles from KILT (10), where missing titles are imputed with BM-25 (11). DPR contexts are imputed (12) if lacking five gold contexts during FiD model pre-training (13). FiD model is fine-tuned using five reranked contexts (14).
• Figure 3: Re3val Inference Pipeline. Reranker concatenates retrieved DPR contexts (1), page titles (2), and query to rerank page titles (3). Contexts retrieved with the top five reranked page titles (4), including BM-25 imputed titles (5), are reranked (6). The top-5 reranked contexts are used to generate an answer (7).
• Figure 4: The decoding process in Re3val involves the utilization of DEARDR PTHL state machine decoding. During decoding, each page is conditionally decoded based on the previous page, as there are instances where multiple page titles are mapped to an answer. Furthermore, a query may have various answers, further influencing the decoding process.