Dense Text Retrieval based on Pretrained Language Models: A Survey

TL;DR

This survey surveys the emergence of dense retrieval driven by pretrained language models, organizing the literature around architecture, training, indexing, and integration to guide both research and deployment. It analyzes cross-encoder and bi-encoder architectures, contrasting their accuracy and efficiency, and details techniques such as negative sampling, data augmentation, and retrieval-augmented pretraining that push performance. The work also covers dense vector indexing via ANNS, hybrid sparse-dense approaches, and pipeline training with rerankers, while highlighting zero-shot robustness and model-based retrieval as active frontiers. Practical contributions include a resource website and open-source codebases to reproduce and extend dense retrieval systems, underscoring the shift toward end-to-end, scalable retrieval pipelines. Overall, the survey provides a thorough, actionable reference for developing PLM-based dense retrieval systems across information retrieval, NLP tasks, and industry applications.

Abstract

Text retrieval is a long-standing research topic on information seeking, where a system is required to return relevant information resources to user's queries in natural language. From classic retrieval methods to learning-based ranking functions, the underlying retrieval models have been continually evolved with the ever-lasting technical innovation. To design effective retrieval models, a key point lies in how to learn the text representation and model the relevance matching. The recent success of pretrained language models (PLMs) sheds light on developing more capable text retrieval approaches by leveraging the excellent modeling capacity of PLMs. With powerful PLMs, we can effectively learn the representations of queries and texts in the latent representation space, and further construct the semantic matching function between the dense vectors for relevance modeling. Such a retrieval approach is referred to as dense retrieval, since it employs dense vectors (a.k.a., embeddings) to represent the texts. Considering the rapid progress on dense retrieval, in this survey, we systematically review the recent advances on PLM-based dense retrieval. Different from previous surveys on dense retrieval, we take a new perspective to organize the related work by four major aspects, including architecture, training, indexing and integration, and summarize the mainstream techniques for each aspect. We thoroughly survey the literature, and include 300+ related reference papers on dense retrieval. To support our survey, we create a website for providing useful resources, and release a code repertory and toolkit for implementing dense retrieval models. This survey aims to provide a comprehensive, practical reference focused on the major progress for dense text retrieval.

Figures (4)

• Figure 1: The illustration for the overall pipeline of an information retrieval system.
• Figure 2: Visual illustration of dual-encoder and cross-encoder architectures.
• Figure 3: Illustration of text representation and similarity search based on product quantization (PQ). Here, we assume that there are $b_1$ centroid sets, each with $b_2$ centroid embeddings. In this representation, an original text vector will be assigned with $b_1$ PQ indices, where each PQ index maps to a centroid embedding from the corresponding centroid set. The quantization-based representation of a text is a vector of $b_1$ PQ indices (corresponding to the $b_1$ nearest centroid embeddings). When evaluating the similarity of a text, we can simply sum the entries from the similarity table with its $b_1$ PQ indices.
• Figure 4: Comparison of the three approaches for pipeline training of retriever and reranker.