Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents

TL;DR

This paper investigates how to allocate reasoning budgets in deep search agents by focusing on listwise reranking within BrowseComp-Plus. It introduces Effective Token Cost ($ETC$) to quantify efficiency across caching and generation costs and systematically analyzes how reranking depth, model size, and reasoning budgets interact. Results show that reranking consistently improves retrieval and end-to-end accuracy, and that moderate reranking often yields similar or better accuracy at substantially lower $ETC$ than increasing search-time reasoning. The findings advocate prioritizing reranking as a cost-effective lever for building scalable deep search systems, with practical guidance on when to invest in reranking versus reasoning. All code is available at the project repository.

Abstract

Deep research agents rely on iterative retrieval and reasoning to answer complex queries, but scaling test-time computation raises significant efficiency concerns. We study how to allocate reasoning budget in deep search pipelines, focusing on the role of listwise reranking. Using the BrowseComp-Plus benchmark, we analyze tradeoffs between model scale, reasoning effort, reranking depth, and total token cost via a novel effective token cost (ETC) metric. Our results show that reranking consistently improves retrieval and end-to-end accuracy, and that moderate reranking often yields larger gains than increasing search-time reasoning, achieving comparable accuracy at substantially lower cost. All our code is available at https://github.com/texttron/BrowseComp-Plus.git

Figures (7)

• Figure 1: Recall@5 improvement vs. the effective cost per one million tokens for reranking depth of $d\in \{10, 20, 50\}$ with oss-20b and oss-120b models under low and medium reasoning efforts.
• Figure 2: Accuracy improvement vs. the effective cost per ten million tokens for oss-20b and oss-120b deep search agents under low, medium, and high reasoning effort and reranking depth of $d\in \{0, 10, 20, 50\}$. In all cases top-5 candidates are passed to the search agent.
• Figure 3: Inference prompt for deep research agents.
• Figure 4: Inference prompt for listwise reranking.
• Figure 5: Inference prompt for evaluating end-to-end answer correctness.