Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical Foundations of DIME: Risk Estimation for Practical Index Selection

Giulio D'Erasmo, Cesare Campagnano, Antonio Mallia, Pierpaolo Brutti, Nicola Tonellotto, Fabrizio Silvestri

TL;DR

The paper tackles the inefficiency of high-dimensional dense embeddings in information retrieval by proposing RDIME, a statistically grounded, per-query dimensionality selection criterion that eliminates the need for grid-search over global dimensionalities. Building on DIME and a modulation-estimator framework, it introduces Kernel DIME to weight contributions from top documents and derives a hard-thresholding rule that identifies which embedding dimensions carry latent query signal. Empirically, RDIME achieves parity with full-dimension baselines and fixed-k pruning while substantially reducing embedding size on average, across multiple models and datasets, without validation-set tuning. The work highlights a path toward adaptive, query-dependent dimension reduction in dense retrieval and discusses future directions for kernel choices and theoretical guarantees.

Abstract

High-dimensional dense embeddings have become central to modern Information Retrieval, but many dimensions are noisy or redundant. Recently proposed DIME (Dimension IMportance Estimation), provides query-dependent scores to identify informative components of embeddings. DIME relies on a costly grid search to select a priori a dimensionality for all the query corpus's embeddings. Our work provides a statistically grounded criterion that directly identifies the optimal set of dimensions for each query at inference time. Experiments confirm achieving parity of effectiveness and reduces embedding size by an average of $\sim50\%$ across different models and datasets at inference time.

Statistical Foundations of DIME: Risk Estimation for Practical Index Selection

TL;DR

The paper tackles the inefficiency of high-dimensional dense embeddings in information retrieval by proposing RDIME, a statistically grounded, per-query dimensionality selection criterion that eliminates the need for grid-search over global dimensionalities. Building on DIME and a modulation-estimator framework, it introduces Kernel DIME to weight contributions from top documents and derives a hard-thresholding rule that identifies which embedding dimensions carry latent query signal. Empirically, RDIME achieves parity with full-dimension baselines and fixed-k pruning while substantially reducing embedding size on average, across multiple models and datasets, without validation-set tuning. The work highlights a path toward adaptive, query-dependent dimension reduction in dense retrieval and discusses future directions for kernel choices and theoretical guarantees.

Abstract

High-dimensional dense embeddings have become central to modern Information Retrieval, but many dimensions are noisy or redundant. Recently proposed DIME (Dimension IMportance Estimation), provides query-dependent scores to identify informative components of embeddings. DIME relies on a costly grid search to select a priori a dimensionality for all the query corpus's embeddings. Our work provides a statistically grounded criterion that directly identifies the optimal set of dimensions for each query at inference time. Experiments confirm achieving parity of effectiveness and reduces embedding size by an average of across different models and datasets at inference time.
Paper Structure (16 sections, 3 theorems, 20 equations, 1 figure, 2 tables)

This paper contains 16 sections, 3 theorems, 20 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. DIME
  4. Top-$k$ Thresholding
  5. Modulation Estimators
  6. Method
  7. IR for Modulation Estimators
  8. DIME as an Estimator of Squared Latent Signal
  9. Experimental Section
  10. Results
  11. Conclusion
  12. Limitations
  13. Appendix
  14. Optimal weights under heteroskedastic noise in IR
  15. Importance of Query-Wise Dimension Selection
  16. ...and 1 more sections

Key Result

Theorem 1

Given the noisy embedding $\bm{q}$, the optimal hard-threshold estimator $\hat{\bm{\theta}}(S^\star)$ of the latent signal $\bm{\theta}$ under $\ell_2$ risk is given by: where

Figures (1)

  • Figure 1: Percentage of dimensions retained per query across three bi-encoders (ANCE, Contriever, TAS-B) and four collections (DL ’19, DL ’20, DL HD, RB ’04) using our risk-based criterion with three DIME variants. Boxplots show median, interquartile range, and outliers, revealing substantial query-dependent variation.

Theorems & Definitions (6)

  • Theorem 1: Hard Thresholding Estimator
  • proof
  • Definition 1: Kernel DIME
  • Theorem 2
  • proof
  • Corollary 1: RDIME: Risk Dimension Importance Estimation