Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximate Message Passing for orthogonally invariant ensembles: Multivariate non-linearities and spectral initialization

Xinyi Zhong, Tianhao Wang, Zhou Fan

Abstract

We study a class of Approximate Message Passing (AMP) algorithms for symmetric and rectangular spiked random matrix models with orthogonally invariant noise. The AMP iterates have fixed dimension $K \geq 1$, a multivariate non-linearity is applied in each AMP iteration, and the algorithm is spectrally initialized with $K$ super-critical sample eigenvectors. We derive the forms of the Onsager debiasing coefficients and corresponding AMP state evolution, which depend on the free cumulants of the noise spectral distribution. This extends previous results for such models with $K=1$ and an independent initialization. Applying this approach to Bayesian principal components analysis, we introduce a Bayes-OAMP algorithm that uses as its non-linearity the posterior mean conditional on all preceding AMP iterates. We describe a practical implementation of this algorithm, where all debiasing and state evolution parameters are estimated from the observed data, and we illustrate the accuracy and stability of this approach in simulations.

Approximate Message Passing for orthogonally invariant ensembles: Multivariate non-linearities and spectral initialization

Abstract

We study a class of Approximate Message Passing (AMP) algorithms for symmetric and rectangular spiked random matrix models with orthogonally invariant noise. The AMP iterates have fixed dimension , a multivariate non-linearity is applied in each AMP iteration, and the algorithm is spectrally initialized with super-critical sample eigenvectors. We derive the forms of the Onsager debiasing coefficients and corresponding AMP state evolution, which depend on the free cumulants of the noise spectral distribution. This extends previous results for such models with and an independent initialization. Applying this approach to Bayesian principal components analysis, we introduce a Bayes-OAMP algorithm that uses as its non-linearity the posterior mean conditional on all preceding AMP iterates. We describe a practical implementation of this algorithm, where all debiasing and state evolution parameters are estimated from the observed data, and we illustrate the accuracy and stability of this approach in simulations.

Paper Structure

This paper contains 60 sections, 16 theorems, 319 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Notational conventions.
  4. Symmetric AMP with independent initialization
  5. Debiasing coefficients.
  6. State Evolution.
  7. Spectral initialization for the symmetric spiked model
  8. Preliminaries on sample eigenvectors and the $R$-transform
  9. Eigenvectors and spectral phase transition.
  10. $R$-transform.
  11. AMP algorithm
  12. Debiasing coefficients.
  13. State Evolution.
  14. Proof idea
  15. Orthogonal AMP for Bayesian PCA
  16. ...and 45 more sections

Key Result

Theorem \oldthetheorem

Suppose Assumptions assump:symW and assump:symindinit hold. For any $T \geq 1$, consider the AMP algorithm (eq:AMPz--eq:AMPu) up to iteration $T$, and define $(U_1,\ldots,U_{T+1},Z_1,\ldots,Z_T,E)$ by the state evolution (eq:SEindinit). Then almost surely as $n \to \infty$,

Figures (4)

  • Figure 1: Estimation errors for AMP iterates $\mathbf{U}_t$ in the symmetric spiked model with $n=4000$, rank-2 signal, and signal prior $U_* \sim \frac{1}{2}\delta_{(0,1)}+\frac{1}{4}\delta_{(\sqrt{2},-1)}+\frac{1}{4}\delta_{(-\sqrt{2},-1)}$. Boxes indicate the $\{25,50,75\}$-percentiles across 50 random trials, and whiskers indicate $1.5 \times \text{inter-quartile range}$. Iteration 0 corresponds to the spectral initialization $\mathbf{U}_0$. The noise spectral distributions are (left) the semicircle law, (middle) $\textnormal{Uniform}[-\sqrt{3},\sqrt{3}]$, and (right) standarized $\textnormal{Beta}(3,1)$.
  • Figure 2: Distributions of iterates $\mathbf{F}_0,\mathbf{F}_1,\mathbf{F}_2$ in the Centered Beta noise example of Figure \ref{['fig:sym:algocombat']}. Shown are histograms of the empirical distributions of the two columns of each iterate $\mathbf{F}_t$ (denoted PC1 and PC2), overlaid with the marginal density of the corresponding coordinate of the state evolution law $\mathcal{N}(\pmb{\mu}_t \cdot U_*,\mathbf{\Sigma}_t)$. This density agrees closely for Bayes-OAMP, whereas a large discrepancy is observed for the state evolution prediction of Gaussian Bayes-AMP.
  • Figure 3: Estimation errors for the sample eigenvector and three AMP algorithms, in a symmetric model with Centered Beta noise spectrum, $n = 4000$, rank-1 signal with prior $U_*\sim \frac{1}{2}\delta_{-1} + \frac{1}{2}\delta_{1}$, and signal strength $\theta$ varying from $0.1$ to $2.0$. The spectral "phase transition" occurs at $\theta=0$.
  • Figure 4: Estimation errors for AMP iterates $\mathbf{U}_t$ and $\mathbf{V}_t$ in the rectangular spiked model with $(m,n)=(3000,4000)$, rank-2 signal, signal priors $U_*,V_* \sim \frac{1}{2}\delta_{(0,1)}+\frac{1}{4}\delta_{(\sqrt{2},-1)}+\frac{1}{4}\delta_{(-\sqrt{2},-1)}$, and signal strengths $(\theta_1,\theta_2)=(2,1.5)$. Iterates $\mathbf{V}_0$ and $\mathbf{U}_1$ correspond to the spectral initializations. The noise spectral distributions are (left) square-root of the Marcenko-Pastur law, (middle) $\textnormal{Uniform}[\sqrt{3/7},2\sqrt{3/7}]$, and (right) the $\sqrt{5/3}\cdot\textnormal{Beta}(3,1)$ distribution.

Theorems & Definitions (34)

  • Theorem \oldthetheorem
  • Remark \oldthetheorem
  • Theorem \oldthetheorem: benaych2011eigenvalues
  • Theorem \oldthetheorem
  • Remark \oldthetheorem
  • Remark \oldthetheorem
  • Theorem \oldthetheorem
  • Remark \oldthetheorem
  • Theorem \oldthetheorem: benaych2012singular
  • Theorem \oldthetheorem
  • ...and 24 more