Improved Sublinear-time Moment Estimation using Weighted Sampling
Anup Bhattacharya, Pinki Pradhan
TL;DR
This work studies the t-th moment estimation problem $S_t=\sum_{a\in A} w(a)^t$ under proportional sampling, aiming for accurate estimates with few queries. It develops sublinear algorithms for $t>1$, matching known upper bounds and establishing tight lower bounds; for $t>1/2$ a sublinear algorithm is achieved while no sublinear algorithm exists for $t\le 1/2$. A key contribution is the moment-density parameter $\rho$, which yields refined upper and lower bounds on sample complexity and captures beyond-worst-case behavior. The paper also analyzes hybrid sampling and proves it provides no worst-case advantage, and it introduces a systematic technique combining an estimate of $W=\sum w(a)$ with a biased estimator for $S_t$ stabilized by concentration tools. Overall, the results give a near-complete picture of the sample complexity landscape for moment estimation with proportional sampling, including $Θ(n^{1-1/t}\ln(1/\delta)/\epsilon^2)$ optimality for $t\ge 2$ and a first sublinear algorithm for $1/2<t<1$.
Abstract
In this work we study the {\it moment estimation} problem using weighted sampling. Given sample access to a set $A$ with $n$ weighted elements, and a parameter $t>0$, we estimate the $t$-th moment of $A$ given as $S_t=\sum_{a\in A} w(a)^t$. For t=1, this is the sum estimation problem. The moment estimation problem along with a number of its variants have been extensively studied in streaming, sublinear and distributed communication models. Despite being well studied, we don't yet have a complete understanding of the sample complexity of the moment estimation problem in the sublinear model and in this work, we make progress on this front. On the algorithmic side, our upper bounds match the known upper bounds for the problem for $t>1$. To the best of our knowledge, no sublinear algorithms were known for this problem for $0<t<1$. We design a sublinear algorithm for this problem for $t>1/2$ and show that no sublinear algorithms exist for $t\leq 1/2$. We prove a $Ω(\frac{n^{1-1/t}\ln 1/δ}{ε^2})$ lower bound for moment estimation for $t>1$, and show optimal sample complexity bound $Θ(\frac{n^{1-1/t}\ln 1/δ}{ε^2})$ for moment estimation for $t\geq 2$. Hence, we obtain a complete understanding of the sample complexity for moment estimation using proportional sampling for $t\geq 2$. We also study the moment estimation problem in the beyond worst-case analysis paradigm and identify a new {\it moment-density} parameter of the input that characterizes the sample complexity of the problem using proportional sampling and derive tight sample complexity bounds with respect to that parameter. We also study the moment estimation problem in the hybrid sampling framework in which one is given additional access to a uniform sampling oracle and show that hybrid sampling framework does not provide any additional gain over the proportional sampling oracle in the worst case.