Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The dynamics of message passing on dense graphs, with applications to compressed sensing

Mohsen Bayati, Andrea Montanari

TL;DR

<3-5 sentence high-level summary> The paper establishes a rigorous foundation for state evolution (SE) in approximate message passing (AMP) algorithms applied to dense sensing matrices in compressed sensing. It proves that, in the large-system limit with Gaussian sensing, the entire AMP dynamics is tracked by a simple one-dimensional recursion for the effective noise variance and yields a decoupled scalar description of coordinate-wise estimates. The authors introduce a novel conditioning technique (Bolthausen’s method) to handle the dense-graph setting with many short loops and to prove asymptotic Gaussianity of iterates, enabling precise performance predictions and connections to MMSE and LASSO. They also discuss universality conjectures, provide concrete examples (linear estimation, soft-thresholded CS, multi-user detection), and generalize the results to the symmetric case, illustrating the broad applicability of SE beyond standard tree-like graph analyses.

Abstract

Approximate message passing algorithms proved to be extremely effective in reconstructing sparse signals from a small number of incoherent linear measurements. Extensive numerical experiments further showed that their dynamics is accurately tracked by a simple one-dimensional iteration termed state evolution. In this paper we provide the first rigorous foundation to state evolution. We prove that indeed it holds asymptotically in the large system limit for sensing matrices with independent and identically distributed gaussian entries. While our focus is on message passing algorithms for compressed sensing, the analysis extends beyond this setting, to a general class of algorithms on dense graphs. In this context, state evolution plays the role that density evolution has for sparse graphs. The proof technique is fundamentally different from the standard approach to density evolution, in that it copes with large number of short loops in the underlying factor graph. It relies instead on a conditioning technique recently developed by Erwin Bolthausen in the context of spin glass theory.

The dynamics of message passing on dense graphs, with applications to compressed sensing

TL;DR

<3-5 sentence high-level summary> The paper establishes a rigorous foundation for state evolution (SE) in approximate message passing (AMP) algorithms applied to dense sensing matrices in compressed sensing. It proves that, in the large-system limit with Gaussian sensing, the entire AMP dynamics is tracked by a simple one-dimensional recursion for the effective noise variance and yields a decoupled scalar description of coordinate-wise estimates. The authors introduce a novel conditioning technique (Bolthausen’s method) to handle the dense-graph setting with many short loops and to prove asymptotic Gaussianity of iterates, enabling precise performance predictions and connections to MMSE and LASSO. They also discuss universality conjectures, provide concrete examples (linear estimation, soft-thresholded CS, multi-user detection), and generalize the results to the symmetric case, illustrating the broad applicability of SE beyond standard tree-like graph analyses.

Abstract

Approximate message passing algorithms proved to be extremely effective in reconstructing sparse signals from a small number of incoherent linear measurements. Extensive numerical experiments further showed that their dynamics is accurately tracked by a simple one-dimensional iteration termed state evolution. In this paper we provide the first rigorous foundation to state evolution. We prove that indeed it holds asymptotically in the large system limit for sensing matrices with independent and identically distributed gaussian entries. While our focus is on message passing algorithms for compressed sensing, the analysis extends beyond this setting, to a general class of algorithms on dense graphs. In this context, state evolution plays the role that density evolution has for sparse graphs. The proof technique is fundamentally different from the standard approach to density evolution, in that it copes with large number of short loops in the underlying factor graph. It relies instead on a conditioning technique recently developed by Erwin Bolthausen in the context of spin glass theory.

Paper Structure

This paper contains 36 sections, 19 theorems, 106 equations.

Table of Contents

  1. Introduction and main results
  2. Main result
  3. Universality
  4. State evolution: the basic intuition
  5. Related literature
  6. Examples
  7. Linear estimation
  8. Compressed sensing via soft thresholding
  9. Multi-User Detection
  10. Proof
  11. The conditioning technique: an informal description
  12. A general result
  13. Corollary of Theorem \ref{['thm:general']}: AMP and Theorem \ref{['thm:main']}
  14. Proof of Theorem \ref{['thm:main']}
  15. Definitions and notations
  16. ...and 21 more sections

Key Result

Theorem 1

Let $\{A(N)\}_{N\ge 0}$ be a sequence of sensing matrices $A\in{\mathbb R}^{n\times N}$ indexed by $N$, with i.i.d. entries $A_{ij}\sim {\sf N}(0,1/n)$, and assume $n/N\to\delta\in (0,\infty)$. Consider further a sequence of signals $\{x_0(N)\}_{N\ge 0}$, whose empirical distributions converge weakl with $X_0\sim p_{X_0}$ and $Z\sim {\sf N}(0,1)$ independent.

Theorems & Definitions (21)

  • Theorem 1
  • Corollary 1: Decoupling principle
  • Theorem 2
  • Lemma 1
  • Theorem 3: SLLN, SLLN2
  • Lemma 2
  • Lemma 3
  • Lemma 4: Stein's Lemma Ste72
  • Lemma 5
  • Lemma 6
  • ...and 11 more