Sum Estimation via Vector Similarity Search

TL;DR

This paper addresses the problem of estimating sums over large vector datasets by introducing a level-based framework that leverages a maximization oracle to retrieve top-k items from exponentially leveled data structures. It proves the estimator is unbiased and establishes a high-probability relative error bound that scales as $\mathcal{O}\left(\sqrt{\frac{\log(1/\delta)}{k}}\right)$ with a logarithmic number of levels, enabling $O(\log n)$ retrieved elements to suffice. The approach is validated across KDE, softmax denominator estimation, and counting tasks on Open Images, Flickr30k, and Amazon Reviews, showing improved accuracy and runtime over traditional top-k or random-sampling baselines. The work also introduces a control-variate term to reduce variance and discusses integration opportunities with HNSW to further boost efficiency, highlighting practical impact for density estimation and large-scale probabilistic modeling.

Abstract

Semantic embeddings to represent objects such as image, text and audio are widely used in machine learning and have spurred the development of vector similarity search methods for retrieving semantically related objects. In this work, we study the sibling task of estimating a sum over all objects in a set, such as the kernel density estimate (KDE) and the normalizing constant for softmax distributions. While existing solutions provably reduce the sum estimation task to acquiring $\mathcal{O}(\sqrt{n})$ most similar vectors, where $n$ is the number of objects, we introduce a novel algorithm that only requires $\mathcal{O}(\log(n))$ most similar vectors. Our approach randomly assigns objects to levels with exponentially-decaying probabilities and constructs a vector similarity search data structure for each level. With the top-$k$ objects from each level, we propose an unbiased estimate of the sum and prove a high-probability relative error bound. We run experiments on OpenImages and Amazon Reviews with a vector similar search implementation to show that our method can achieve lower error using less computational time than existing reductions. We show results on applications in estimating densities, computing softmax denominators, and counting the number of vectors within a ball.

Figures (5)

• Figure 1: An illustration of the proof of the bound on the size of $U$. The expected number of elements at every level is shown in blue. The minimum with $k$ is shown in dark blue. There are $\ell^*$ levels with more than $k$ elements in expectation, and the remaining levels have less than $2k$ elements in expectation.
• Figure 2: (a) Median relative error for KDE and various bandwidths illustrate the flat to peaky spectrum. (b) Comparison of our analytical upper bound and an empirical 95% confidence interval, computed using tail probabilities of the $\text{Binomial}(100,0.95)$ distribution.
• Figure 3: Trade-off plot comparing our proposed algorithm against the baselines. For the y-axis, we report the median relative error at the task parameter value (e.g., bandwidth, temperature) that maximizes it. Random and TopK have a median relative error close to 1, so it is cutoff in the plot. We include 95% confidence intervals for the median (both x-axis and y-axis) using tail probabilities of the $\text{Binomial}(30, 0.5)$ distribution.
• Figure 4: Median relative error for various settings as a function of the setting parameter (bandwidth, radius, temperature) to illustrate the flat to peaky spectrum. For the counting tasks, the lines are not plotted when more than 75% of the runs have $F=0$ which gives indeterminate relative error. In the other cases, the median is only taken with respect to runs with $F \neq 0$
• Figure 5: The recall as a function of the level for different values of $k$ for different settings. We only plot levels where the number of points at the level is greater than $k$. The error bars are two standard errors.

Theorems & Definitions (29)

• Proposition 3.1
• proof
• Theorem 3.2
• proof
• Lemma 3.3
• Theorem 3.4
• Lemma 3.5
• proof
• Lemma 3.6
• proof