Multivector Reranking in the Era of Strong First-Stage Retrievers
Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini
TL;DR
Multivector retrievers achieve strong retrieval effectiveness but rely on expensive token-level gathering. The authors replace token-level gathering with a document-level first stage based on Learned Sparse Retrieval (LSR), forming a compact candidate set for a dense multivector reranker that uses a $MaxSim$ late-interaction to produce the final ranking. The paper studies four multivector compression schemes and two lightweight reranking optimizations, demonstrating that a two-stage pipeline can reach up to $24\times$ speedups with comparable or superior retrieval quality, in a fully reproducible end-to-end implementation. The work highlights a practical path toward scalable multivector retrieval by marrying LSR-first-stage efficiency with targeted reranking, significantly reducing latency while preserving effectiveness in realistic settings.
Abstract
Learned multivector representations power modern search systems with strong retrieval effectiveness, but their real-world use is limited by the high cost of exhaustive token-level retrieval. Therefore, most systems adopt a \emph{gather-and-refine} strategy, where a lightweight gather phase selects candidates for full scoring. However, this approach requires expensive searches over large token-level indexes and often misses the documents that would rank highest under full similarity. In this paper, we reproduce several state-of-the-art multivector retrieval methods on two publicly available datasets, providing a clear picture of the current multivector retrieval field and observing the inefficiency of token-level gathering. Building on top of that, we show that replacing the token-level gather phase with a single-vector document retriever -- specifically, a learned sparse retriever (LSR) -- produces a smaller and more semantically coherent candidate set. This recasts the gather-and-refine pipeline into the well-established two-stage retrieval architecture. As retrieval latency decreases, query encoding with two neural encoders becomes the dominant computational bottleneck. To mitigate this, we integrate recent inference-free LSR methods, demonstrating that they preserve the retrieval effectiveness of the dual-encoder pipeline while substantially reducing query encoding time. Finally, we investigate multiple reranking configurations that balance efficiency, memory, and effectiveness, and we introduce two optimization techniques that prune low-quality candidates early. Empirical results show that these techniques improve retrieval efficiency by up to 1.8$\times$ with no loss in quality. Overall, our two-stage approach achieves over $24\times$ speedup over the state-of-the-art multivector retrieval systems, while maintaining comparable or superior retrieval quality.