Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universality of Approximate Message Passing algorithms and tensor networks

Tianhao Wang, Xinyi Zhong, Zhou Fan

TL;DR

The paper establishes universal state-evolution results for Approximate Message Passing (AMP) algorithms beyond classical Gaussian and rotationally invariant ensembles by introducing generalized Wigner and generalized invariant matrix classes. A two-step proof framework reduces AMP universality to the universality of tensor-network evaluations of diagonal tensors, culminating in an asymptotic freeness-like property for diagonal observables. The results cover symmetric and rectangular settings, including matrices with heavy-tailed entries, heteroskedastic variances, and structured transforms (subsampled Hadamard/DFT), and extend to both white-noise and invariant ensembles with precise covariance prescriptions. Practical implications include universal phase transitions in compressed sensing and robust AMP performance for data matrices with complex variance structures, as well as computationally efficient algorithms leveraging fast transforms. The work broadens the scope of AMP analysis and provides a foundation for dynamical mean-field descriptions of first-order iterative methods on a wider class of random matrices.

Abstract

Approximate Message Passing (AMP) algorithms provide a valuable tool for studying mean-field approximations and dynamics in a variety of applications. Although these algorithms are often first derived for matrices having independent Gaussian entries or satisfying rotational invariance in law, their state evolution characterizations are expected to hold over larger universality classes of random matrix ensembles. We develop several new results on AMP universality. For AMP algorithms tailored to independent Gaussian entries, we show that their state evolutions hold over broadly defined generalized Wigner and white noise ensembles, including matrices with heavy-tailed entries and heterogeneous entrywise variances that may arise in data applications. For AMP algorithms tailored to rotational invariance in law, we show that their state evolutions hold over delocalized sign-and-permutation-invariant matrix ensembles that have a limit distribution over the diagonal, including sensing matrices composed of subsampled Hadamard or Fourier transforms and diagonal operators. We establish these results via a simplified moment-method proof, reducing AMP universality to the study of products of random matrices and diagonal tensors along a tensor network. As a by-product of our analyses, we show that the aforementioned matrix ensembles satisfy a notion of asymptotic freeness with respect to such tensor networks, which parallels usual definitions of freeness for traces of matrix products.

Universality of Approximate Message Passing algorithms and tensor networks

TL;DR

The paper establishes universal state-evolution results for Approximate Message Passing (AMP) algorithms beyond classical Gaussian and rotationally invariant ensembles by introducing generalized Wigner and generalized invariant matrix classes. A two-step proof framework reduces AMP universality to the universality of tensor-network evaluations of diagonal tensors, culminating in an asymptotic freeness-like property for diagonal observables. The results cover symmetric and rectangular settings, including matrices with heavy-tailed entries, heteroskedastic variances, and structured transforms (subsampled Hadamard/DFT), and extend to both white-noise and invariant ensembles with precise covariance prescriptions. Practical implications include universal phase transitions in compressed sensing and robust AMP performance for data matrices with complex variance structures, as well as computationally efficient algorithms leveraging fast transforms. The work broadens the scope of AMP analysis and provides a foundation for dynamical mean-field descriptions of first-order iterative methods on a wider class of random matrices.

Abstract

Approximate Message Passing (AMP) algorithms provide a valuable tool for studying mean-field approximations and dynamics in a variety of applications. Although these algorithms are often first derived for matrices having independent Gaussian entries or satisfying rotational invariance in law, their state evolution characterizations are expected to hold over larger universality classes of random matrix ensembles. We develop several new results on AMP universality. For AMP algorithms tailored to independent Gaussian entries, we show that their state evolutions hold over broadly defined generalized Wigner and white noise ensembles, including matrices with heavy-tailed entries and heterogeneous entrywise variances that may arise in data applications. For AMP algorithms tailored to rotational invariance in law, we show that their state evolutions hold over delocalized sign-and-permutation-invariant matrix ensembles that have a limit distribution over the diagonal, including sensing matrices composed of subsampled Hadamard or Fourier transforms and diagonal operators. We establish these results via a simplified moment-method proof, reducing AMP universality to the study of products of random matrices and diagonal tensors along a tensor network. As a by-product of our analyses, we show that the aforementioned matrix ensembles satisfy a notion of asymptotic freeness with respect to such tensor networks, which parallels usual definitions of freeness for traces of matrix products.
Paper Structure (22 sections, 35 theorems, 302 equations)

This paper contains 22 sections, 35 theorems, 302 equations.

Table of Contents

  1. Introduction
  2. Contributions
  3. Notation
  4. Main results
  5. Universality of AMP algorithms for symmetric matrices
  6. Tensor networks and strategy of proof
  7. Universality of AMP algorithms for rectangular matrices
  8. Applications
  9. Proofs for symmetric matrices
  10. Universality for generalized Wigner matrices
  11. Universality for symmetric generalized invariant matrices
  12. Universality of AMP via polynomial approximation
  13. Discussion
  14. Density of polynomials
  15. Sufficient conditions for generalized invariance
  16. ...and 7 more sections

Key Result

Theorem 2.4

Let $\mathbf{W} \in \mathbb{R}^{n \times n}$ be a generalized Wigner matrix with variance profile $\mathbf{S}$, and let $\mathbf{u}_1,\mathbf{f}_1,\ldots,\mathbf{f}_k$ be independent of $\mathbf{W}$ and satisfy Assumption assump:ufconvergence. Suppose that Let $\{b_{ts}\}$ and $\{\bm{\Sigma}_t\}$ be defined by the GOE prescriptions (eq:GOEbSigma) and (eq:GOEb), where each matrix $\bm{\Sigma}_t$ i

Theorems & Definitions (81)

  • Remark 2.2
  • Definition 2.3
  • Theorem 2.4
  • Definition 2.5
  • Definition 2.6
  • Proposition 2.7
  • Theorem 2.8
  • Remark 2.9
  • Remark 2.10
  • Definition 2.11
  • ...and 71 more