Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Retrieve Passages without Supervision

Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, Amir Globerson

TL;DR

This work tackles the problem of open-domain QA retrieval with minimal labeled data by introducing Spider, a dense retriever pretrained in a self-supervised, contrastive manner. Spider exploits cross-passage recurring spans to generate pseudo question–passage pairs, using a query transformation that balances lexical and semantic cues and a span-filtering strategy to focus on meaningful spans. The results show substantial gains in unsupervised retrieval, strong cross-dataset generalization, and notable improvements when used to initialize supervised DPR training, with a simple BM25–Spider hybrid further boosting performance. Overall, the study demonstrates that well-designed pretraining for dense retrievers can drastically reduce the need for labeled data in ODQA, enabling effective retrieval across domains and datasets while remaining computationally efficient relative to end-to-end approaches.

Abstract

Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our "recurring span retrieval" approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseudo queries and relevant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreover, a hybrid retriever over Spider and BM25 improves over both, and is often competitive with DPR models, which are trained on tens of thousands of examples. Last, notable gains are observed when using Spider as an initialization for supervised training.

Learning to Retrieve Passages without Supervision

TL;DR

This work tackles the problem of open-domain QA retrieval with minimal labeled data by introducing Spider, a dense retriever pretrained in a self-supervised, contrastive manner. Spider exploits cross-passage recurring spans to generate pseudo question–passage pairs, using a query transformation that balances lexical and semantic cues and a span-filtering strategy to focus on meaningful spans. The results show substantial gains in unsupervised retrieval, strong cross-dataset generalization, and notable improvements when used to initialize supervised DPR training, with a simple BM25–Spider hybrid further boosting performance. Overall, the study demonstrates that well-designed pretraining for dense retrievers can drastically reduce the need for labeled data in ODQA, enabling effective retrieval across domains and datasets while remaining computationally efficient relative to end-to-end approaches.

Abstract

Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our "recurring span retrieval" approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseudo queries and relevant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreover, a hybrid retriever over Spider and BM25 improves over both, and is often competitive with DPR models, which are trained on tens of thousands of examples. Last, notable gains are observed when using Spider as an initialization for supervised training.
Paper Structure (41 sections, 3 equations, 2 figures, 8 tables)

This paper contains 41 sections, 3 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Our Model: Spider
  4. Pretraining: Recurring Span Retrieval
  5. Query Transformation
  6. Span Filtering
  7. Training
  8. Hybrid Dense-Sparse Retrieval
  9. Experimental Setup
  10. Datasets
  11. Baselines
  12. Evaluation Settings
  13. Unsupervised Setting
  14. Cross-Dataset Generalization
  15. Supervised Setting
  16. ...and 26 more sections

Figures (2)

  • Figure 1: Top-$k$ retrieval accuracy of various unsupervised methods (solid lines) on the test set of Natural Questions (NQ). DPR (dotted) is supervised (trained on NQ) and given for reference.
  • Figure 2: An example of our pretraining approach: Given a document $\mathcal{D}$ (e.g. the article "Aaron" in Wikipedia), we take two passages that contain a recurring span $S$. One of them is transformed into a short query (left) $q'$ using a random window surrounding $S$, in which $S$ is either kept (top) or removed (bottom). The second passage is then considered the target for retrieval $p^+$, while a random passage from $\mathcal{D}$ that does not contain $S$ is considered the negative $p^-$ (right). Each batch is comprised of multiple such examples, and the pretraining task is to select the passage $p_i^+$ for each query $q'_i$ (solid line) from the passages of all examples (in-batch negatives; dashed lines).