Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Re2G: Retrieve, Rerank, Generate

Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, Alfio Gliozzo

TL;DR

The paper tackles knowledge-intensive NLP by integrating retrieval, a learnable reranker, and generation in a single Re^2G framework. It introduces a four-phase training regime plus online knowledge distillation to align retrieval with reranking, enabling effective end-to-end learning. Empirically, Re^2G achieves large relative gains across four KILT tasks (slot filling, QA, fact checking, dialog) and supports ensembling initial retrieval sources (BM25 and neural methods). The work demonstrates the value of explicit reranking and retrieval fusion for improving generation, with practical impact on deploying knowledge-grounded NLP systems and releasing open-source code for replication.

Abstract

As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger. However, for tasks that require a large amount of knowledge, non-parametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements. Recent models such as RAG and REALM have introduced retrieval into conditional generation. These models incorporate neural initial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation. Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval. To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker, and generation using only ground truth on the target sequence output. We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact-checking, and dialog, with relative gains of 9% to 34% over the previous state-of-the-art on the KILT leaderboard. We make our code available as open source at https://github.com/IBM/kgi-slot-filling/tree/re2g.

Re2G: Retrieve, Rerank, Generate

TL;DR

The paper tackles knowledge-intensive NLP by integrating retrieval, a learnable reranker, and generation in a single Re^2G framework. It introduces a four-phase training regime plus online knowledge distillation to align retrieval with reranking, enabling effective end-to-end learning. Empirically, Re^2G achieves large relative gains across four KILT tasks (slot filling, QA, fact checking, dialog) and supports ensembling initial retrieval sources (BM25 and neural methods). The work demonstrates the value of explicit reranking and retrieval fusion for improving generation, with practical impact on deploying knowledge-grounded NLP systems and releasing open-source code for replication.

Abstract

As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger. However, for tasks that require a large amount of knowledge, non-parametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements. Recent models such as RAG and REALM have introduced retrieval into conditional generation. These models incorporate neural initial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation. Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval. To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker, and generation using only ground truth on the target sequence output. We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact-checking, and dialog, with relative gains of 9% to 34% over the previous state-of-the-art on the KILT leaderboard. We make our code available as open source at https://github.com/IBM/kgi-slot-filling/tree/re2g.
Paper Structure (21 sections, 3 equations, 5 figures, 7 tables)

This paper contains 21 sections, 3 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Reranker
  5. Training
  6. Reranking Training
  7. End-to-End Training
  8. Inference
  9. Experiments
  10. Retrieval
  11. Ablations
  12. Analysis
  13. Slot filling error analysis
  14. Conclusions
  15. Hyperparameters
  16. ...and 6 more sections

Figures (5)

  • Figure 1: KILT tasks of slot filling, question answering, fact checking and dialog
  • Figure 2: RAG Architecture
  • Figure 3: $\text{Re}^2\text{G}$ Architecture
  • Figure 4: Interaction Model Reranker
  • Figure 5: Representation Model for Initial Retrieval