Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Model Architectures in Information Retrieval

Zhichao Xu, Fengran Mo, Zhiqi Huang, Crystina Zhang, Puxuan Yu, Bei Wang, Jimmy Lin, Vivek Srikumar

Abstract

The period from 2019 to the present marks one of the most significant paradigm shifts in information retrieval (IR) and natural language processing (NLP), culminating in the emergence of powerful large language models (LLMs) from 2022 onward. Methods based on pretrained encoder-only architectures (e.g., BERT) as well as decoder-only generative LLMs have outperformed many earlier approaches, demonstrating particularly strong performance in zero-shot scenarios and complex reasoning tasks. This survey examines the evolution of model architectures in IR, with a focus on two key aspects: backbone models for feature extraction and end-to-end system architectures for relevance estimation. To maintain analytical clarity, we deliberately separate architectural design from training methodologies, enabling a focused examination of structural innovations in IR systems. We trace the progression from traditional term-based retrieval models to modern neural approaches, highlighting the transformative impact of transformer-based architectures and subsequent LLM developments. The survey concludes with a forward-looking discussion of open challenges and emerging research directions, including architectural optimization for efficiency and scalability, robust handling of multimodal and multilingual data, and adaptation to novel application domains such as autonomous search agents, which may represent the next paradigm in IR.

A Survey of Model Architectures in Information Retrieval

Abstract

The period from 2019 to the present marks one of the most significant paradigm shifts in information retrieval (IR) and natural language processing (NLP), culminating in the emergence of powerful large language models (LLMs) from 2022 onward. Methods based on pretrained encoder-only architectures (e.g., BERT) as well as decoder-only generative LLMs have outperformed many earlier approaches, demonstrating particularly strong performance in zero-shot scenarios and complex reasoning tasks. This survey examines the evolution of model architectures in IR, with a focus on two key aspects: backbone models for feature extraction and end-to-end system architectures for relevance estimation. To maintain analytical clarity, we deliberately separate architectural design from training methodologies, enabling a focused examination of structural innovations in IR systems. We trace the progression from traditional term-based retrieval models to modern neural approaches, highlighting the transformative impact of transformer-based architectures and subsequent LLM developments. The survey concludes with a forward-looking discussion of open challenges and emerging research directions, including architectural optimization for efficiency and scalability, robust handling of multimodal and multilingual data, and adaptation to novel application domains such as autonomous search agents, which may represent the next paradigm in IR.

Paper Structure

This paper contains 94 sections, 7 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Terminology
  3. Task Definition and Evaluation.
  4. The Multi-Stage "Retrieve-then-Rerank" Architecture.
  5. Query and Document.
  6. Disentangling Model Architecture from Training Strategies.
  7. Traditional IR Models
  8. Boolean Model.
  9. Vector Space Model.
  10. Probabilistic Model.
  11. Statistical Language Model.
  12. Learning-to-Rank Model Architectures
  13. Feature Engineering in LTR.
  14. Pointwise, Pairwise, and Listwise Approaches with ML Models
  15. Neural LTR Models
  16. ...and 79 more sections

Figures (3)

  • Figure 1: High-level overview of the survey structure and scope.
  • Figure 2: Illustration on neural ranking models. Brown boxes indicate uncontextualized word embeddings (e.g., Word2vec).
  • Figure 3: Illustration on transformer-based retrieval and reranking models. Yellow boxes indicate pretrained Transformers (e.g., BERT). Query text, embeddings, and associated weights are color-coded in orange, whereas document representations are color-coded in blue.