CRISP: Clustering Multi-Vector Representations for Denoising and Pruning

TL;DR

CRISP tackles the efficiency gap of multi-vector IR by training token representations to be inherently clusterable, allowing aggressive pruning without relying on post-hoc clustering. It integrates clustering into end-to-end training via K-means to produce cluster centroids that represent token groups, reducing the number of vectors while preserving or improving retrieval quality on BEIR. The approach yields substantial compression (up to 11x document and 7.9x query with small quality loss) and demonstrates denoising effects, outperforming post-hoc clustering across tasks. This elevates the practicality of multi-vector retrieval by delivering robust, compact representations with minimal performance penalties.

Abstract

Multi-vector models, such as ColBERT, are a significant advancement in neural information retrieval (IR), delivering state-of-the-art performance by representing queries and documents by multiple contextualized token-level embeddings. However, this increased representation size introduces considerable storage and computational overheads which have hindered widespread adoption in practice. A common approach to mitigate this overhead is to cluster the model's frozen vectors, but this strategy's effectiveness is fundamentally limited by the intrinsic clusterability of these embeddings. In this work, we introduce CRISP (Clustered Representations with Intrinsic Structure Pruning), a novel multi-vector training method which learns inherently clusterable representations directly within the end-to-end training process. By integrating clustering into the training phase rather than imposing it post-hoc, CRISP significantly outperforms post-hoc clustering at all representation sizes, as well as other token pruning methods. On the BEIR retrieval benchmarks, CRISP achieves a significant rate of ~3x reduction in the number of vectors while outperforming the original unpruned model. This indicates that learned clustering effectively denoises the model by filtering irrelevant information, thereby generating more robust multi-vector representations. With more aggressive clustering, CRISP achieves an 11x reduction in the number of vectors with only a $3.6\%$ quality loss.

Figures (3)

• Figure 1: Illustration of Tail Pruning, where the last $k$ vectors (here $k=4$) are retained (shown in black), and others are shown in white.
• Figure 2: Illustration of K-Spacing, where every $k$-th vector (here $k=2$) is retained (shown in black), and others are shown in white.
• Figure 3: Illustration of Clustering Pruning: Original vectors (top row, styled by assigned cluster) are mapped to their respective cluster centroids (bottom row). Here, 3 clusters (C1-C3) are used, distinguished by white, black, and gray fills. The selected embeddings for each cluster are calculated dynamically so they need not be adjacent.