Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eigenstructure inference for high-dimensional covariance with generalized shrinkage inverse-Wishart prior

Seongmin Kim, Kwangmin Lee, Sewon Park, Jaeyong Lee

TL;DR

The paper tackles the problem of unreliable eigenstructure estimation in high-dimensional covariance matrices by introducing the generalized shrinkage inverse-Wishart (gSIW) prior, which allows componentwise shrinkage of eigenvalues under a spiked covariance model. It derives the conjugate posterior for the spectral components, develops a Gibbs sampler for efficient computation, and provides three practical strategies (WAIC, GR, IC$_{p3}$) to select the number of spikes $k$, with WAIC recommended. Theoretical results establish explicit posterior convergence rates for both eigenvalues and eigenvectors under the gSIW prior, including regimes for equal and unequal shrinkage parameters $a_i$, and demonstrate how the posterior concentrates near the true eigenstructure. Simulation studies and a MNIST data application show that gSIW improves eigenvalue estimation (especially for top spikes) with tighter credible intervals, while achieving competitive eigenvector performance, indicating effective high-dimensional eigenstructure inference in practice.

Abstract

In multivariate statistics, estimating the covariance matrix is essential for understanding the interdependence among variables. In high-dimensional settings, where the number of covariates increases with the sample size, it is well known that the eigenstructure of the sample covariance matrix is inconsistent. The inverse-Wishart prior, a standard choice for covariance estimation in Bayesian inference, also suffers from posterior inconsistency. To address the issue of eigenvalue dispersion in high-dimensional settings, the shrinkage inverse-Wishart (SIW) prior has recently been proposed. Despite its conceptual appeal and empirical success, the asymptotic justification for the SIW prior has remained limited. In this paper, we propose a generalized shrinkage inverse-Wishart (gSIW) prior for high-dimensional covariance modeling. By extending the SIW framework, the gSIW prior accommodates a broader class of prior distributions and facilitates the derivation of theoretical properties under specific parameter choices. In particular, under the spiked covariance assumption, we establish the asymptotic behavior of the posterior distribution for both eigenvalues and eigenvectors by directly evaluating the posterior expectations for two sets of parameter choices. This direct evaluation provides insights into the large-sample behavior of the posterior that cannot be obtained through general posterior asymptotic theorems. Finally, simulation studies illustrate that the proposed prior provides accurate estimation of the eigenstructure, particularly for spiked eigenvalues, achieving narrower credible intervals and higher coverage probabilities compared to existing methods. For spiked eigenvectors, the performance is generally comparable to that of competing approaches, including the sample covariance.

Eigenstructure inference for high-dimensional covariance with generalized shrinkage inverse-Wishart prior

TL;DR

The paper tackles the problem of unreliable eigenstructure estimation in high-dimensional covariance matrices by introducing the generalized shrinkage inverse-Wishart (gSIW) prior, which allows componentwise shrinkage of eigenvalues under a spiked covariance model. It derives the conjugate posterior for the spectral components, develops a Gibbs sampler for efficient computation, and provides three practical strategies (WAIC, GR, IC) to select the number of spikes , with WAIC recommended. Theoretical results establish explicit posterior convergence rates for both eigenvalues and eigenvectors under the gSIW prior, including regimes for equal and unequal shrinkage parameters , and demonstrate how the posterior concentrates near the true eigenstructure. Simulation studies and a MNIST data application show that gSIW improves eigenvalue estimation (especially for top spikes) with tighter credible intervals, while achieving competitive eigenvector performance, indicating effective high-dimensional eigenstructure inference in practice.

Abstract

In multivariate statistics, estimating the covariance matrix is essential for understanding the interdependence among variables. In high-dimensional settings, where the number of covariates increases with the sample size, it is well known that the eigenstructure of the sample covariance matrix is inconsistent. The inverse-Wishart prior, a standard choice for covariance estimation in Bayesian inference, also suffers from posterior inconsistency. To address the issue of eigenvalue dispersion in high-dimensional settings, the shrinkage inverse-Wishart (SIW) prior has recently been proposed. Despite its conceptual appeal and empirical success, the asymptotic justification for the SIW prior has remained limited. In this paper, we propose a generalized shrinkage inverse-Wishart (gSIW) prior for high-dimensional covariance modeling. By extending the SIW framework, the gSIW prior accommodates a broader class of prior distributions and facilitates the derivation of theoretical properties under specific parameter choices. In particular, under the spiked covariance assumption, we establish the asymptotic behavior of the posterior distribution for both eigenvalues and eigenvectors by directly evaluating the posterior expectations for two sets of parameter choices. This direct evaluation provides insights into the large-sample behavior of the posterior that cannot be obtained through general posterior asymptotic theorems. Finally, simulation studies illustrate that the proposed prior provides accurate estimation of the eigenstructure, particularly for spiked eigenvalues, achieving narrower credible intervals and higher coverage probabilities compared to existing methods. For spiked eigenvectors, the performance is generally comparable to that of competing approaches, including the sample covariance.

Paper Structure

This paper contains 12 sections, 6 theorems, 41 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Generalized shrinkage inverse-Wishart prior and its posterior
  3. Notation
  4. Generalized shrinkage inverse-Wishart prior
  5. Sampling Method
  6. Choice of the number of spiked eigenvalues $k$
  7. Main Results
  8. Simulation Studies
  9. Case 1
  10. Case 2
  11. Real Data
  12. Conclusion

Key Result

Lemma 3.1

Under model eq:model and prior eq:SIW, assume that conditions $A1-A5$ hold. Let $\epsilon > 0$, and define $\kappa = \min\limits_{l<k}(a_{l+1} - a_l) \cdot \min\limits_{l<k}\log(\dfrac{\lambda_{0,l}}{\lambda_{0,l+1}})$. Suppose that $\dfrac{np}{a_n}\prec \dfrac{\lambda_{0,k}}{n\lambda_{0,1}}\dfrac{\ where $D_\epsilon$ is defined in eq:D_epsilon.

Figures (3)

  • Figure 1: The images of selected $50$ MNIST samples labeled as $7$.
  • Figure 2: The images after dimensionality reduction of the selected $50$ MNIST samples labeled as $7$ using different model selection criteria.
  • Figure 3: The first 50 eigenvalues of the sample covariance matrix.

Theorems & Definitions (6)

  • Lemma 3.1
  • Lemma 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Theorem 3.5
  • Corollary 3.6