Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energy

Michael Celentano

TL;DR

This work develops a Sudakov-Fernique post-AMP inequality to study local optimization landscapes around AMP iterates in GOE settings, enabling precise geometric analysis of non-convex TAP free energies in $ ext{Z}_2$-synchronization and SK sampling. By conditioning on the entire AMP trajectory, the authors reduce a high-dimensional, non-convex Hessian problem to a tractable analysis at the state-evolution fixed point, ultimately proving local strong convexity of the TAP free energy near late AMP iterates. The unified approach yields a simpler proof of local convexity for the TAP functional in Celentano–Fan–Mei 2021 and extends to the related TAP energy used by Alaoui–Montanari–Selke 2022, confirming polynomial-time sampling of the SK Gibbs measure in the easy regime. The results advance understanding of the two-stage non-convex optimization paradigm, showing that first-order methods can be guided into regions of strong convexity where classical convex optimization guarantees apply.

Abstract

In many problems in modern statistics and machine learning, it is often of interest to establish that a first order method on a non-convex risk function eventually enters a region of parameter space in which the risk is locally convex. We derive an asymptotic comparison inequality, which we call the Sudakov-Fernique post-AMP inequality, which, in a certain class of problems involving a GOE matrix, is able to probe properties of an optimization landscape locally around the iterates of an approximate message passing (AMP) algorithm. As an example of its use, we provide a new, and arguably simpler, proof of some of the results of Celentano et al. (2021), which establishes that the so-called TAP free energy in the $\mathbb{Z}_2$-synchronization problem is locally convex in the region to which AMP converges. We further prove a conjecture of El Alaoui et al. (2022) involving the local convexity of a related but distinct TAP free energy, which, as a consequence, confirms that their algorithm efficiently samples from the Sherrington-Kirkpatrick Gibbs measure throughout the "easy" regime.

Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energy

TL;DR

This work develops a Sudakov-Fernique post-AMP inequality to study local optimization landscapes around AMP iterates in GOE settings, enabling precise geometric analysis of non-convex TAP free energies in -synchronization and SK sampling. By conditioning on the entire AMP trajectory, the authors reduce a high-dimensional, non-convex Hessian problem to a tractable analysis at the state-evolution fixed point, ultimately proving local strong convexity of the TAP free energy near late AMP iterates. The unified approach yields a simpler proof of local convexity for the TAP functional in Celentano–Fan–Mei 2021 and extends to the related TAP energy used by Alaoui–Montanari–Selke 2022, confirming polynomial-time sampling of the SK Gibbs measure in the easy regime. The results advance understanding of the two-stage non-convex optimization paradigm, showing that first-order methods can be guided into regions of strong convexity where classical convex optimization guarantees apply.

Abstract

In many problems in modern statistics and machine learning, it is often of interest to establish that a first order method on a non-convex risk function eventually enters a region of parameter space in which the risk is locally convex. We derive an asymptotic comparison inequality, which we call the Sudakov-Fernique post-AMP inequality, which, in a certain class of problems involving a GOE matrix, is able to probe properties of an optimization landscape locally around the iterates of an approximate message passing (AMP) algorithm. As an example of its use, we provide a new, and arguably simpler, proof of some of the results of Celentano et al. (2021), which establishes that the so-called TAP free energy in the -synchronization problem is locally convex in the region to which AMP converges. We further prove a conjecture of El Alaoui et al. (2022) involving the local convexity of a related but distinct TAP free energy, which, as a consequence, confirms that their algorithm efficiently samples from the Sherrington-Kirkpatrick Gibbs measure throughout the "easy" regime.
Paper Structure (21 sections, 11 theorems, 117 equations)

This paper contains 21 sections, 11 theorems, 117 equations.

Table of Contents

  1. Introduction
  2. Background on $\mathbb{Z}_2$-synchronization and the Sherrington-Kirkpatrick model
  3. Overview of results
  4. Related literature
  5. Notation
  6. The Sudakov-Fernique post-AMP inequality
  7. Sudakov-Fernique post-AMP: the general case
  8. Sudakov-Fernique post-AMP in $\mathbb{Z}_2$-synchronization
  9. Local convexity of the TAP free energy
  10. Proof of Theorem \ref{['thm:local-strong-cvx']}
  11. Reduction to optimization on Wasserstein space
  12. Dimensionality reduction of Wasserstein space
  13. Reduction to analysis at state evolution fixed point
  14. Analysis of $\mathsf{F}_{{\mathsf x}}^{(2)}$
  15. Proof of the Sudakov-Fernique post-AMP inequality
  16. ...and 6 more sections

Key Result

Proposition 1

Let ${\boldsymbol W} \sim {\rm GOE}(n)$, ${\boldsymbol \xi} \sim {\sf N}({\boldsymbol 0},{\mathbf I}_n)$, and for ${\boldsymbol v} \in {\mathbb R}^n$, let ${\boldsymbol g}({\boldsymbol v}) := \| {\boldsymbol v} \| {\boldsymbol \xi}$. Then for any $t \in {\mathbb R}$, closed set $K \subset {\mathbb R (Here ${\boldsymbol v} \in {\mathbb R}^n$ and ${\boldsymbol u} \in {\mathbb R}^m$ for some non-nega

Theorems & Definitions (18)

  • Proposition 1: Sudakov-Fernique inequality
  • Proposition 2: State evolution
  • Theorem 1: Sudakov-Fernique post-AMP
  • Proposition 3: State evolution
  • Corollary 1: Sudakov-Fernique post-AMP in $\mathbb{Z}_2$-synchronization
  • Theorem 2: Local convexity
  • Remark 1
  • Lemma 1
  • Lemma 2: Existence of solutions to Eq. \ref{['eq:B-def']}
  • proof : Proof of Lemma \ref{['lem:exist-unique']}
  • ...and 8 more