Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GAR-meets-RAG Paradigm for Zero-Shot Information Retrieval

Daman Arora, Anush Kini, Sayak Ray Chowdhury, Nagarajan Natarajan, Gaurav Sinha, Amit Sharma

TL;DR

<3-5 sentence high-level summary> The paper tackles zero-shot information retrieval by uniting generation-augmented retrieval (GAR) and retrieval-augmented generation (RAG) into a recurrence framework. It introduces Rewrite-Retrieve-Rerank (RRR), a three-stage, iterative process where a RAG-generated rewrite informs GAR retrieval, and GAR-derived results feed back into subsequent rewrites, topped off by a neural re-ranker to boost precision. Using a relevance model to filter and calibrate retrievals and a trimmed augmentation strategy, RRR achieves state-of-the-art results on BEIR (notably Recall@100 and nDCG@10 on 6 of 8 datasets) and strong TREC-DL performance in the zero-shot setting. The work demonstrates the value of a feedback loop between rewrite and retrieval for improving recall without target-domain training data, while outlining limitations around model size and inference cost and highlighting directions for distillation and compact re-ranking models.

Abstract

Given a query and a document corpus, the information retrieval (IR) task is to output a ranked list of relevant documents. Combining large language models (LLMs) with embedding-based retrieval models, recent work shows promising results on the zero-shot retrieval problem, i.e., no access to labeled data from the target domain. Two such popular paradigms are generation-augmented retrieval or GAR (generate additional context for the query and then retrieve), and retrieval-augmented generation or RAG (retrieve relevant documents as context and then generate answers). The success of these paradigms hinges on (i) high-recall retrieval models, which are difficult to obtain in the zero-shot setting, and (ii) high-precision (re-)ranking models which typically need a good initialization. In this work, we propose a novel GAR-meets-RAG recurrence formulation that overcomes the challenges of existing paradigms. Our method iteratively improves retrieval (via GAR) and rewrite (via RAG) stages in the zero-shot setting. A key design principle is that the rewrite-retrieval stages improve the recall of the system and a final re-ranking stage improves the precision. We conduct extensive experiments on zero-shot passage retrieval benchmarks, BEIR and TREC-DL. Our method establishes a new state-of-the-art in the BEIR benchmark, outperforming previous best results in Recall@100 and nDCG@10 metrics on 6 out of 8 datasets, with up to 17% relative gains over the previous best.

GAR-meets-RAG Paradigm for Zero-Shot Information Retrieval

TL;DR

<3-5 sentence high-level summary> The paper tackles zero-shot information retrieval by uniting generation-augmented retrieval (GAR) and retrieval-augmented generation (RAG) into a recurrence framework. It introduces Rewrite-Retrieve-Rerank (RRR), a three-stage, iterative process where a RAG-generated rewrite informs GAR retrieval, and GAR-derived results feed back into subsequent rewrites, topped off by a neural re-ranker to boost precision. Using a relevance model to filter and calibrate retrievals and a trimmed augmentation strategy, RRR achieves state-of-the-art results on BEIR (notably Recall@100 and nDCG@10 on 6 of 8 datasets) and strong TREC-DL performance in the zero-shot setting. The work demonstrates the value of a feedback loop between rewrite and retrieval for improving recall without target-domain training data, while outlining limitations around model size and inference cost and highlighting directions for distillation and compact re-ranking models.

Abstract

Given a query and a document corpus, the information retrieval (IR) task is to output a ranked list of relevant documents. Combining large language models (LLMs) with embedding-based retrieval models, recent work shows promising results on the zero-shot retrieval problem, i.e., no access to labeled data from the target domain. Two such popular paradigms are generation-augmented retrieval or GAR (generate additional context for the query and then retrieve), and retrieval-augmented generation or RAG (retrieve relevant documents as context and then generate answers). The success of these paradigms hinges on (i) high-recall retrieval models, which are difficult to obtain in the zero-shot setting, and (ii) high-precision (re-)ranking models which typically need a good initialization. In this work, we propose a novel GAR-meets-RAG recurrence formulation that overcomes the challenges of existing paradigms. Our method iteratively improves retrieval (via GAR) and rewrite (via RAG) stages in the zero-shot setting. A key design principle is that the rewrite-retrieval stages improve the recall of the system and a final re-ranking stage improves the precision. We conduct extensive experiments on zero-shot passage retrieval benchmarks, BEIR and TREC-DL. Our method establishes a new state-of-the-art in the BEIR benchmark, outperforming previous best results in Recall@100 and nDCG@10 metrics on 6 out of 8 datasets, with up to 17% relative gains over the previous best.
Paper Structure (33 sections, 2 equations, 1 figure, 12 tables)

This paper contains 33 sections, 2 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Background and Notation
  3. Zero-shot IR Problem:
  4. Metrics:
  5. Retrieval model:
  6. Generative (language) model:
  7. Relevance model:
  8. Proposed Method: GAR meets RAG
  9. Formulation
  10. Algorithm: RRR
  11. Evaluation Setup
  12. (A) Datasets and metrics:
  13. (B) Compared techniques:
  14. (C) Implementation details:
  15. (D) Hyperparameter selection:
  16. ...and 18 more sections

Figures (1)

  • Figure 1: Proposed RRR method for zero-shot Information Retrieval. We implement the rewrite, filtering, and re-rank stages (colored boxes) via a pre-trained LLMs (in our evaluations, we use GPT-3.5-Turbo and GPT-4 models). For the retrieval stage, we use BM25. Details in Section \ref{['sec:method']}.