Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fourier Analysis of Iterative Algorithms

Chris Jones, Lucas Pesenti

TL;DR

A general class of nonlinear iterative algorithms which includes power iteration, belief propagation and approximate message passing, and many forms of gradient descent are studied, using Boolean Fourier analysis to analyze these algorithms as low-degree polynomials in the entries of the input matrix.

Abstract

We study a general class of nonlinear iterative algorithms which includes power iteration, belief propagation and approximate message passing, and many forms of gradient descent. When the input is a random matrix with i.i.d. entries, we use Boolean Fourier analysis to analyze these algorithms as low-degree polynomials in the entries of the input matrix. Each symmetrized Fourier character represents all monomials with a certain shape as specified by a small graph, which we call a Fourier diagram. We prove fundamental asymptotic properties of the Fourier diagrams: over the randomness of the input, all diagrams with cycles are negligible; the tree-shaped diagrams form a basis of asymptotically independent Gaussian vectors; and, when restricted to the trees, iterative algorithms exactly follow an idealized Gaussian dynamic. We use this to prove a state evolution formula, giving a "complete" asymptotic description of the algorithm's trajectory. The restriction to tree-shaped monomials mirrors the assumption of the cavity method, a 40-year-old non-rigorous technique in statistical physics which has served as one of the most important techniques in the field. We demonstrate how to implement cavity method derivations by 1) restricting the iteration to its tree approximation, and 2) observing that heuristic cavity method-type arguments hold rigorously on the simplified iteration. Our proofs use combinatorial arguments similar to the trace method from random matrix theory. Finally, we push the diagram analysis to a number of iterations that scales with the dimension $n$ of the input matrix, proving that the tree approximation still holds for a simple variant of power iteration all the way up to $n^{Ω(1)}$ iterations.

Fourier Analysis of Iterative Algorithms

TL;DR

A general class of nonlinear iterative algorithms which includes power iteration, belief propagation and approximate message passing, and many forms of gradient descent are studied, using Boolean Fourier analysis to analyze these algorithms as low-degree polynomials in the entries of the input matrix.

Abstract

We study a general class of nonlinear iterative algorithms which includes power iteration, belief propagation and approximate message passing, and many forms of gradient descent. When the input is a random matrix with i.i.d. entries, we use Boolean Fourier analysis to analyze these algorithms as low-degree polynomials in the entries of the input matrix. Each symmetrized Fourier character represents all monomials with a certain shape as specified by a small graph, which we call a Fourier diagram. We prove fundamental asymptotic properties of the Fourier diagrams: over the randomness of the input, all diagrams with cycles are negligible; the tree-shaped diagrams form a basis of asymptotically independent Gaussian vectors; and, when restricted to the trees, iterative algorithms exactly follow an idealized Gaussian dynamic. We use this to prove a state evolution formula, giving a "complete" asymptotic description of the algorithm's trajectory. The restriction to tree-shaped monomials mirrors the assumption of the cavity method, a 40-year-old non-rigorous technique in statistical physics which has served as one of the most important techniques in the field. We demonstrate how to implement cavity method derivations by 1) restricting the iteration to its tree approximation, and 2) observing that heuristic cavity method-type arguments hold rigorously on the simplified iteration. Our proofs use combinatorial arguments similar to the trace method from random matrix theory. Finally, we push the diagram analysis to a number of iterations that scales with the dimension of the input matrix, proving that the tree approximation still holds for a simple variant of power iteration all the way up to iterations.
Paper Structure (51 sections, 60 theorems, 144 equations, 1 figure, 1 table)

This paper contains 51 sections, 60 theorems, 144 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Fourier diagrams.
  4. Asymptotic diagram analysis.
  5. The cavity method.
  6. Taking the tree approximation farther.
  7. Conclusion.
  8. Related work
  9. Comparison with prior work.
  10. Analyzing algorithms as low-degree polynomials.
  11. Statistical physics and the cavity method.
  12. Belief propagation and AMP.
  13. Organization of the paper
  14. Acknowledgments.
  15. Preliminaries
  16. ...and 36 more sections

Key Result

Theorem 1.1

For any $k\ge 0$ independent of $n$, for all connected Fourier diagrams $\alpha_1, \dots, \alpha_k$ and $i_1, \ldots, i_k\in [n]$ (allowing repetitions in $\alpha_j$ and $i_j$), where for any connected Fourier diagram $\alpha$ and $i\in [n]$,

Figures (1)

  • Figure 1: Diagram representation of the cavity messages $m^t_{i\to j}$. Each cavity diagram in the asymptotic cavity diagram representation of $m^t_{i\to j}$ is a tree rooted at $i$.

Theorems & Definitions (138)

  • Theorem 1.1: Classification theorem; see \ref{['thm:classification']}
  • Theorem 1.2: Tree approximation of GFOMs; see \ref{['thm:state-evolution']}
  • Theorem 1.3: General state evolution; see \ref{['thm:general-state-evolution']}
  • Definition 1.4: $\,\overset{\infty}{=}\,$, informal version of \ref{['def:asymptotic-equality']}
  • Theorem 1.5: Equivalence of BP and AMP; see \ref{['thm:bp-amp']}
  • Theorem 1.6: Asymptotic independence of incoming messages; see \ref{['thm:cavity-formal']}
  • Theorem 1.7: Asymptotic state of AMP; see \ref{['thm:amp-state']}
  • Theorem 1.8: See \ref{['thm:power-iteration']}
  • Definition 2.2: Convergence of random vectors
  • Lemma 2.3: Method of moments billingsley
  • ...and 128 more