LEMUR: Learned Multi-Vector Retrieval

TL;DR

LEMUR addresses the latency barrier of multi-vector retrieval by learning a latent representation that turns MaxSim-based queries into a single-vector inner product problem. It reframes multi-vector similarity as a supervised, token-level regression task and then reduces inference to single-vector ANNS in a learned latent space, enabling fast retrieval with existing ANN libraries. Across six BEIR datasets and ViDoRe visual retrieval, Lemur consistently delivers large speedups while maintaining or improving recall, and it demonstrates robustness across diverse multi-vector embeddings beyond ColBERTv2. The approach offers practical benefits for scalable IR systems by lowering latency, reducing memory usage through lower-dimensional latent vectors, and facilitating easy integration with standard single-vector ANN engines.

Abstract

Multi-vector representations generated by late interaction models, such as ColBERT, enable superior retrieval quality compared to single-vector representations in information retrieval applications. In multi-vector retrieval systems, both queries and documents are encoded using one embedding for each token, and similarity between queries and documents is measured by the MaxSim similarity measure. However, the improved recall of multi-vector retrieval comes at the expense of significantly increased latency. This necessitates designing efficient approximate nearest neighbor search (ANNS) algorithms for multi-vector search. In this work, we introduce LEMUR, a simple-yet-efficient framework for multi-vector similarity search. LEMUR consists of two consecutive problem reductions: We first formulate multi-vector similarity search as a supervised learning problem that can be solved using a one-hidden-layer neural network. Second, we reduce inference under this model to single-vector similarity search in its latent space, which enables the use of existing single-vector ANNS methods for speeding up retrieval. In addition to performance evaluation on ColBERTv2 embeddings, we evaluate LEMUR on embeddings generated by modern multi-vector text models and multi-vector visual document retrieval models. LEMUR is an order of magnitude faster than earlier multi-vector similarity search methods.

Figures (17)

• Figure 1: A schematic overview of the query process (for indexing, see Sec. \ref{['sec:framework']}) in the LEMUR framework: The latent representations $\psi(x)$ of the token-level embeddings $x\in X$ are retrieved from the hidden layer of an MLP trained to estimate the MaxSim similarities between a query and each document. The single-vector representation $\Psi(X)$ is obtained by pooling these latent representations. The $k^\prime$ most similar documents to $\Psi(X)$ are retrieved using an ANNS index. The final top-$k$ documents are selected by evaluating the exact MaxSim similarities to these $k^\prime$ documents.
• Figure 2: Ablation study on the effect of the hidden layer size $d'$ on the performance of Lemur. Left: Comparison of $\text{recall100@}k'$ for three values of $d'$ as a function of the candidate set size $k'$. Right: Comparison of the end-to-end-performance comparison between different values of $d'$ with ANNS and reranking included. While larger values of $d'$ can yield more accurate estimates, the end-to-end performance gap is narrower due to increased ANNS complexity, yielding diminishing returns.
• Figure 3: Ablation study on the effect of using ANNS in Lemur. ANNS significantly speeds up retrieval at recall levels $< 0.95$.
• Figure 4: End-to-end performance comparison using ColBERTv2 embeddings on the HotpotQA (left) and MS MARCO (right) datasets. On both datasets, Lemur is significantly faster than the baseline methods.
• Figure 5: End-to-end performance comparison using four different modern multi-vector text models on the SCIDOCS dataset. On all datasets, Lemur is significantly faster than the baseline methods, while especially MUVERA struggles on the non-ColBERTv2 models.