Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Empirical Spectral Distributions for Random Tensor Product Models

Simona Diaconu

TL;DR

The paper studies the empirical spectral distribution (ESD) of covariance matrices generated from the random tensor product model and proves almost-sure convergence to the Marchenko–Pastur law in the regime $d/n^{1/2}=o(1)$ under symmetry assumptions and slowly growing subgaussian norms. It develops a three-step framework based on the Stieltjes transform, concentration, and analytic continuation, with a truncation-based approach to control dependent features. It extends the isotropic MP-limit to nonisotropic covariances via a universality argument, deriving a fixed-point characterization of the limiting Stieltjes transform and correcting a key step in prior work. The results broaden the applicability of MP-type limits to structured random matrices arising from tensor-product feature constructions, with potential relevance to other dependent-entry models through concentration-driven arguments.

Abstract

In statistics, assuming samples are independent is reasonable. However, this property can fail to hold for the features, a distinction that has led to several lines of work aiming to remove the latter assumption of independence present in the early literature, while preserving the original conclusions. Empirical spectral distributions of covariance matrices are key for understanding the data, and their almost sure convergence is oftentimes desirable. The random tensor product model, $X=(x_{i_1}x_{i_2}...x_{i_d})_{1 \leq i_1<...<i_d \leq n}$ for $x_1,x_2,\hspace{0.05cm}...\hspace{0.05cm},x_n$ i.i.d., introduced by the machine learning community, has a dependence structure for its features far from trivial and has been studied in recent years. When $x_1 \in \mathbb{R}, \mathbb{E}[x_1^4]<\infty, \frac{d}{n^{1/3}}=o(1),$ the empirical spectral distributions of the covariance matrices were proved to converge almost surely to Marchenko-Pastur laws in the random matrix theory regime. This work extends this result to the range $\frac{d}{n^{1/2}}=~o(1)$ when $x_1$ is symmetric with a subgaussian norm slowly growing in $n$ (the aforesaid range arises naturally, and the result failing when $\frac{d}{n^{1/2}} \to \infty$ appears to be a plausible claim) and shows that similarly to the case with independent features, the almost sure convergence holds under more general conditions on the covariance structure than the isotropic case. The latter result provides a means of deriving convergence for empirical spectral distributions of random matrices, applicable to other models as well so long as their entries exhibit a certain degree of concentration.

On Empirical Spectral Distributions for Random Tensor Product Models

TL;DR

The paper studies the empirical spectral distribution (ESD) of covariance matrices generated from the random tensor product model and proves almost-sure convergence to the Marchenko–Pastur law in the regime under symmetry assumptions and slowly growing subgaussian norms. It develops a three-step framework based on the Stieltjes transform, concentration, and analytic continuation, with a truncation-based approach to control dependent features. It extends the isotropic MP-limit to nonisotropic covariances via a universality argument, deriving a fixed-point characterization of the limiting Stieltjes transform and correcting a key step in prior work. The results broaden the applicability of MP-type limits to structured random matrices arising from tensor-product feature constructions, with potential relevance to other dependent-entry models through concentration-driven arguments.

Abstract

In statistics, assuming samples are independent is reasonable. However, this property can fail to hold for the features, a distinction that has led to several lines of work aiming to remove the latter assumption of independence present in the early literature, while preserving the original conclusions. Empirical spectral distributions of covariance matrices are key for understanding the data, and their almost sure convergence is oftentimes desirable. The random tensor product model, for i.i.d., introduced by the machine learning community, has a dependence structure for its features far from trivial and has been studied in recent years. When the empirical spectral distributions of the covariance matrices were proved to converge almost surely to Marchenko-Pastur laws in the random matrix theory regime. This work extends this result to the range when is symmetric with a subgaussian norm slowly growing in (the aforesaid range arises naturally, and the result failing when appears to be a plausible claim) and shows that similarly to the case with independent features, the almost sure convergence holds under more general conditions on the covariance structure than the isotropic case. The latter result provides a means of deriving convergence for empirical spectral distributions of random matrices, applicable to other models as well so long as their entries exhibit a certain degree of concentration.
Paper Structure (11 sections, 13 theorems, 272 equations)

This paper contains 11 sections, 13 theorems, 272 equations.

Table of Contents

  1. Introduction
  2. Revised 3-Step Method
  3. Expectations via Truncations
  4. Tail Expectation
  5. Power Series Expansion
  6. High Moments
  7. Fixing Shared Degrees
  8. Beyond the Isotropic Case
  9. A Crucial Identity
  10. Swapping Random Variables
  11. Auxiliary Results

Key Result

Theorem 1

Suppose $Z \in \mathbb{R}^{N \times p}$ has i.i.d. columns with distributions given by (zmoddel) with $d<n^{1/2},$ for some $C=C(n) \geq 2.$ If (modelz) holds, then the empirical spectral distribution of $S=\frac{1}{p}ZZ^T$ converges weakly to the Marchenko-Pastur law $F_{\gamma},$ given by (mplaws), almost surely.

Theorems & Definitions (22)

  • Theorem 1
  • Lemma 1
  • proof
  • Theorem 2
  • Proposition 1
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Theorem 3
  • ...and 12 more