Hutchinson's Estimator is Bad at Kronecker-Trace-Estimation
Raphael A. Meyer, Haim Avron
TL;DR
The paper addresses trace estimation for PSD matrices accessible only through Kronecker-matrix-vector products, revealing that the natural Kronecker extension of Hutchinson’s estimator suffers exponential dependence on the Kronecker order $k$ in the worst case. By deriving an exact variance formula involving partial trace and partial transpose operators, the authors show that real-valued Kronecker queries incur a $3^k$-type growth, while complex-valued queries can reduce this to $2^k$, with further improvements for small base dimension $d$ and specialized distributions. They establish tight lower bounds and show that rank-one random matrices can be estimated much faster on average, and they demonstrate that exact trace recovery is possible for Kronecker-structured matrices with only $kd+1$ queries. The results illuminate the intrinsic limitations of the Kronecker-Hutchinson approach and suggest avenues for faster or hybrid algorithms that blend Hutchinson-type estimators with alternative techniques, potentially mitigating the exponential dependence on $k$ in practice. Overall, the work provides a rigorous, nuanced understanding of trace estimation in tensor-structured linear algebra settings and lays groundwork for improved methods when exploiting Kronecker structure.
Abstract
We study the problem of estimating the trace of a matrix $\mathbf{A}$ that can only be accessed through Kronecker-matrix-vector products. That is, for any Kronecker-structured vector $\mathrm{x} = \otimes_{i=1}^k \mathrm{x}_i$, we can compute $\mathbf{A}\mathrm{x}$. We focus on the natural generalization of Hutchinson's Estimator to this setting, proving tight rates for the number of matrix-vector products this estimator needs to find a $(1\pm\varepsilon)$ approximation to the trace of $\mathbf{A}$. We find an exact equation for the variance of the estimator when using a Kronecker of Gaussian vectors, revealing an intimate relationship between Hutchinson's Estimator, the partial trace operator, and the partial transpose operator. Using this equation, we show that when using real vectors, in the worst case, this estimator needs $O(\frac{3^k}{\varepsilon^2})$ products to recover a $(1\pm\varepsilon)$ approximation of the trace of any PSD $\mathbf{A}$, and a matching lower bound for certain PSD $\mathbf{A}$. However, when using complex vectors, this can be exponentially improved to $Θ(\frac{2^k}{\varepsilon^2})$. Further, if the $\mathrm{x}_i$ vectors are low-dimensional and if we instead build $\mathrm{x}$ as the Kronecker product of (scaled) random unit vectors on the complex sphere, then as few as $\frac{1.33^k}{\varepsilon^2}$ samples suffice. We show that Hutchinson's Estimator converges slowest when $\mathbf{A}$ itself also has Kronecker structure. We conclude with some theoretical evidence suggesting that, by combining Hutchinson's Estimator with other techniques, it may be possible to avoid the exponential dependence on $k$.