Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Text Ranking in Deep Research

Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton

TL;DR

A selection of key findings and best practices for IR text ranking methods in the deep research setting are reproduced and it is found that agent-issued queries typically follow web-search-style syntax; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval.

Abstract

Deep research has emerged as an important task that aims to address hard queries through extensive open-web exploration. To tackle it, most prior work equips large language model (LLM)-based agents with opaque web search APIs, enabling agents to iteratively issue search queries, retrieve external evidence, and reason over it. Despite search's essential role in deep research, black-box web search APIs hinder systematic analysis of search components, leaving the behaviour of established text ranking methods in deep research largely unclear. To fill this gap, we reproduce a selection of key findings and best practices for IR text ranking methods in the deep research setting. In particular, we examine their effectiveness from three perspectives: (i) retrieval units (documents vs. passages), (ii) pipeline configurations (different retrievers, re-rankers, and re-ranking depths), and (iii) query characteristics (the mismatch between agent-issued queries and the training queries of text rankers). We perform experiments on BrowseComp-Plus, a deep research dataset with a fixed corpus, evaluating 2 open-source agents, 5 retrievers, and 3 re-rankers across diverse setups. We find that agent-issued queries typically follow web-search-style syntax (e.g., quoted exact matches), favouring lexical, learned sparse, and multi-vector retrievers; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval; re-ranking is highly effective; translating agent-issued queries into natural-language questions significantly bridges the query mismatch.

Revisiting Text Ranking in Deep Research

TL;DR

A selection of key findings and best practices for IR text ranking methods in the deep research setting are reproduced and it is found that agent-issued queries typically follow web-search-style syntax; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval.

Abstract

Deep research has emerged as an important task that aims to address hard queries through extensive open-web exploration. To tackle it, most prior work equips large language model (LLM)-based agents with opaque web search APIs, enabling agents to iteratively issue search queries, retrieve external evidence, and reason over it. Despite search's essential role in deep research, black-box web search APIs hinder systematic analysis of search components, leaving the behaviour of established text ranking methods in deep research largely unclear. To fill this gap, we reproduce a selection of key findings and best practices for IR text ranking methods in the deep research setting. In particular, we examine their effectiveness from three perspectives: (i) retrieval units (documents vs. passages), (ii) pipeline configurations (different retrievers, re-rankers, and re-ranking depths), and (iii) query characteristics (the mismatch between agent-issued queries and the training queries of text rankers). We perform experiments on BrowseComp-Plus, a deep research dataset with a fixed corpus, evaluating 2 open-source agents, 5 retrievers, and 3 re-rankers across diverse setups. We find that agent-issued queries typically follow web-search-style syntax (e.g., quoted exact matches), favouring lexical, learned sparse, and multi-vector retrievers; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval; re-ranking is highly effective; translating agent-issued queries into natural-language questions significantly bridges the query mismatch.
Paper Structure (19 sections, 2 equations, 1 figure, 13 tables)

This paper contains 19 sections, 2 equations, 1 figure, 13 tables.

Table of Contents

  1. Introduction
  2. Task definition
  3. Methodology
  4. Research questions and experimental design
  5. Experimental setup
  6. Deep research agents
  7. Text ranking methods to be reproduced
  8. Dataset.
  9. Passage corpus construction
  10. Query-to-question (Q2Q) reformulation
  11. Evaluation.
  12. Implementation details.
  13. Results and Discussions
  14. Sanity check for reproduction
  15. Retrievers on passage and document corpora
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Heatmap from a grid search on BrowseComp-Plus using the original full queries (not end-to-end), showing the effectiveness (evaluated by evidence judgments) of BM25 under different hyperparameter settings. The red $\times$ denotes the default parameter setting following chen2025browsecomp, while the green $+$ denotes the best parameter setting found by the grid search. The lighter the colour, the higher the retrieval performance.