Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, Shaoping Ma

TL;DR

This paper introduces RepBERT, a BERT-based, fixed-length contextual embedding model for first-stage retrieval that uses inner-product scores to rank documents. By encoding documents offline and querying online, RepBERT achieves state-of-the-art first-stage performance on MS MARCO Passage Ranking with efficiency on par with bag-of-words methods. The work analyzes training dynamics, recalls, and reranking interactions, highlighting benefits and mismatches when integrating semantic and exact-match signals, and demonstrates that combining RepBERT with traditional exact-match retrievers yields further gains. Overall, RepBERT shows the feasibility and value of representation-focused neural methods for scalable, high-quality initial retrieval.

Abstract

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.

RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

TL;DR

This paper introduces RepBERT, a BERT-based, fixed-length contextual embedding model for first-stage retrieval that uses inner-product scores to rank documents. By encoding documents offline and querying online, RepBERT achieves state-of-the-art first-stage performance on MS MARCO Passage Ranking with efficiency on par with bag-of-words methods. The work analyzes training dynamics, recalls, and reranking interactions, highlighting benefits and mismatches when integrating semantic and exact-match signals, and demonstrates that combining RepBERT with traditional exact-match retrievers yields further gains. Overall, RepBERT shows the feasibility and value of representation-focused neural methods for scalable, high-quality initial retrieval.

Abstract

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.

Paper Structure

This paper contains 19 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RepBERT
  4. Model Architectures
  5. Training
  6. Experiment
  7. Dataset
  8. Baselines
  9. First-Stage Retrieval
  10. Settings
  11. Discussion
  12. Rerank based on RepBERT
  13. Settings
  14. Discussion
  15. Mismatch
  16. ...and 4 more sections

Figures (2)

  • Figure 1: At different depths, the recall of the first-stage retrieval method and the reranking accuracy of BERT Large. Dataset: MS MARCO dev.
  • Figure 2: For a certain depth, the average proportion of retrieved documents that are also in the official top-1000 candidates provided by MS MARCO. Dataset: MS MARCO dev.