CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale

TL;DR

CoRECT addresses the challenge of evaluating embedding compression for dense retrieval by introducing a controlled, large-scale framework that jointly considers corpus complexity and model-specific behavior. It combines the CoRE dataset with BeIR, four embedding models, and eight compression methods to enable 40+ experimental configurations, using batchwise embedding generation, per-batch compression, and cosine retrieval evaluated with standard IR metrics. The study shows that corpus size and retrieval granularity significantly impact performance and that there is no universal best compression method; method effectiveness is highly model-dependent, though non-learned quantization often offers robust size reductions with minimal loss. The work provides practical guidance for choosing compression techniques in real-world, large-scale retrieval systems and makes the data and code openly available for reproducible, cross-model evaluation.

Abstract

Dense retrieval systems have proven to be effective across various benchmarks, but require substantial memory to store large search indices. Recent advances in embedding compression show that index sizes can be greatly reduced with minimal loss in ranking quality. However, existing studies often overlook the role of corpus complexity -- a critical factor, as recent work shows that both corpus size and document length strongly affect dense retrieval performance. In this paper, we introduce CoRECT (Controlled Retrieval Evaluation of Compression Techniques), a framework for large-scale evaluation of embedding compression methods, supported by a newly curated dataset collection. To demonstrate its utility, we benchmark eight representative types of compression methods. Notably, we show that non-learned compression achieves substantial index size reduction, even on up to 100M passages, with statistically insignificant performance loss. However, selecting the optimal compression method remains challenging, as performance varies across models. Such variability highlights the necessity of CoRECT to enable consistent comparison and informed selection of compression methods. All code, data, and results are available on GitHub and HuggingFace.

Figures (4)

• Figure 1: The CoRECT framework.
• Figure 2: Recall@100 for Snowflake V2 across increasing corpus sizes in different compression settings. Stars indicate significantly lower recall than the blue baseline (one-sided Wilcoxon test, p=0.05). Recall decreases with higher corpus size and granularity. Quantization (middle) outperforms vector truncation (left).
• Figure 3: Average Recall@100 across BeIR datasets for different compression methods. The x-values of some compression methods have been shifted slightly for better visibility. The light gray columns in the background mark the compression ratio to which the methods belong. Shapes indicate different types of compression techniques. Some methods have been combined with vector truncation to achieve the desired compression ratio as indicated in the legend. The top-performing measures vary depending on the model.
• Figure 4: NDCG@10 for Snowflake V2 on the CoRE passage corpora. In contrast to Recall@100, NDCG@10 remains stable with increasing corpus size.