Your Dense Retriever is Secretly an Expeditious Reasoner

TL;DR

Dense retrievers excel at fast similarity search but falter on reasoning-heavy queries. AdaQR introduces a hybrid framework with a Dense Reasoner that imitates LLM reasoning in embedding space and a Reasoner Router that adaptively routes queries to either dense rewriting or full LLM reasoning, balancing efficiency and accuracy. On BRIGHT, AdaQR achieves an average $7.24\%$ improvement in retrieval performance while reducing reasoning cost by $28.12\%$ across multiple dense retrievers and LLMs, with domain-specific effects observed for LLM and dense rewrites. This approach offers a practical path to scalable, reasoning-aware retrieval in large-scale systems by leveraging latent transformations in embedding space plus adaptive decision-making.

Abstract

Dense retrievers enhance retrieval by encoding queries and documents into continuous vectors, but they often struggle with reasoning-intensive queries. Although Large Language Models (LLMs) can reformulate queries to capture complex reasoning, applying them universally incurs significant computational cost. In this work, we propose Adaptive Query Reasoning (AdaQR), a hybrid query rewriting framework. Within this framework, a Reasoner Router dynamically directs each query to either fast dense reasoning or deep LLM reasoning. The dense reasoning is achieved by the Dense Reasoner, which performs LLM-style reasoning directly in the embedding space, enabling a controllable trade-off between efficiency and accuracy. Experiments on large-scale retrieval benchmarks BRIGHT show that AdaQR reduces reasoning cost by 28% while preserving-or even improving-retrieval performance by 7%.

Figures (11)

• Figure 1: Left: PCA-reduced visualization of embedding transformation, where each arrow denotes the shift from the original query embedding to its reasoned counterpart, with most following systematic and structured trajectories. Right: AdaQR on the ReasonIR-8B dense retriever yields substantial gains in retrieval performance and query rewriting efficiency on BRIGHT benchmark.
• Figure 2: Distributions of MRL scores for LLM Reasoners, with each point representing the score computed for a given LLM Reasoner and dense retriever pair.
• Figure 3: Overview of AdaQR: the Reasoner Router directs each input query through the oracle anchor to either the LLM Reasoner or the Dense Reasoner for reasoning-based query rewriting. The Dense Reasoner is constructed via adjacent pre-training and fine-tuning, enabling it to efficiently imitate the reasoning behavior of the LLM in the embedding space.
• Figure 4: Retrieval performance improvement and reasoning cost reduction on the BRIGHT held-out test set achieved by AdaQR compared to LLM-based reasoning.
• Figure 5: Ablation results after removing the Dense Reasoner and Reasoner Router components.