Estimating the Spectral Density of Large Implicit Matrices

TL;DR

This work addresses the challenge of estimating the spectral density of large implicit matrices by casting spectrum inquiries as generalized traces $tr(F(A))$ and employing unbiased randomized Chebyshev expansions. The method combines two key ideas: unbiased estimators of Chebyshev polynomials built from independent matrix-vector estimates and a stochastic trace estimator (Skilling-Hutchinson), augmented with a von Mises kernel to smooth the spectrum and control Gibbs phenomena. It also develops a variance-reduction mechanism via control variates and provides extensive empirical validation on Kneser graphs and classic random-matrix ensembles (Wigner, Wishart, and Wishart+Wigner), showing accurate density recovery under noisy matvecs. The approach yields a practical framework for diagnosing high-dimensional optimization landscapes and dynamic systems where the Hessian or related matrices are large and only accessible implicitly. This paves the way for spectrum-aware analysis in machine learning and statistical modeling with scalable, unbiased estimators.

Abstract

Many important problems are characterized by the eigenvalues of a large matrix. For example, the difficulty of many optimization problems, such as those arising from the fitting of large models in statistics and machine learning, can be investigated via the spectrum of the Hessian of the empirical loss function. Network data can be understood via the eigenstructure of a graph Laplacian matrix using spectral graph theory. Quantum simulations and other many-body problems are often characterized via the eigenvalues of the solution space, as are various dynamic systems. However, naive eigenvalue estimation is computationally expensive even when the matrix can be represented; in many of these situations the matrix is so large as to only be available implicitly via products with vectors. Even worse, one may only have noisy estimates of such matrix vector products. In this work, we combine several different techniques for randomized estimation and show that it is possible to construct unbiased estimators to answer a broad class of questions about the spectra of such implicit matrices, even in the presence of noise. We validate these methods on large-scale problems in which graph theory and random matrix theory provide ground truth.

Figures (4)

• Figure 1: Three von Mises densities are shown with different parameters. On the left they are shown "natively" on the unit circle. On the right they have been projected down to $(-1,1)$. Note that the measure correction can result in boundary effects such as the large density values near $-1$ and $1$.
• Figure 2: The estimated and true spectrum of the Kneser graph $K(23, 11)$. This estimate was generated with $\kappa = 1000$.
• Figure 3: The estimated and true spectral densities of six matrices are shown. These matrices are convex sums of Wishart and Wigner matrices, so their spectra are known in closed form to provide ground truth for the randomized estimation. Estimates were produced using 500 samples with ${\kappa = 5000}$. Although some estimates exhibit variance due to boundary effects, the overall differences are small.
• Figure 4: An example of computing a useful statistic of a large matrix via its spectral density. Here is plotted theoretical and empirical estimates of the index (fraction of negative eigenvalues) of a Wishart+Wigner matrix, as a function of the Wigner fraction. The empirical data here are violin plots computed via the bootstrap over 100 Monte Carlo estimates of the mean spectral density.

Theorems & Definitions (20)

• Proposition 2.1
• proof
• Lemma 3.1
• proof
• Lemma 3.2
• proof
• Proposition 3.1
• proof
• Proposition 3.2
• proof