Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stable Algorithms Lower Bounds for Estimation

Xifan Yu, Ilias Zadik

Abstract

In this work, we show that for all statistical estimation problems, a natural MMSE instability (discontinuity) condition implies the failure of stable algorithms, serving as a version of OGP for estimation tasks. Using this criterion, we establish separations between stable and polynomial-time algorithms for the following MMSE-unstable tasks (i) Planted Shortest Path, where Dijkstra's algorithm succeeds, (ii) random Parity Codes, where Gaussian elimination succeeds, and (iii) Gaussian Subset Sum, where lattice-based methods succeed. For all three, we further show that all low-degree polynomials are stable, yielding separations against low-degree methods and a new method to bound the low-degree MMSE. In particular, our technique highlights that MMSE instability is a common feature for Shortest Path and the noiseless Parity Codes and Gaussian subset sum. Last, we highlight that our work places rigorous algorithmic footing on the long-standing physics belief that first-order phase transitions--which in this setting translates to MMSE-instability impose fundamental limits on classes of efficient algorithms.

Stable Algorithms Lower Bounds for Estimation

Abstract

In this work, we show that for all statistical estimation problems, a natural MMSE instability (discontinuity) condition implies the failure of stable algorithms, serving as a version of OGP for estimation tasks. Using this criterion, we establish separations between stable and polynomial-time algorithms for the following MMSE-unstable tasks (i) Planted Shortest Path, where Dijkstra's algorithm succeeds, (ii) random Parity Codes, where Gaussian elimination succeeds, and (iii) Gaussian Subset Sum, where lattice-based methods succeed. For all three, we further show that all low-degree polynomials are stable, yielding separations against low-degree methods and a new method to bound the low-degree MMSE. In particular, our technique highlights that MMSE instability is a common feature for Shortest Path and the noiseless Parity Codes and Gaussian subset sum. Last, we highlight that our work places rigorous algorithmic footing on the long-standing physics belief that first-order phase transitions--which in this setting translates to MMSE-instability impose fundamental limits on classes of efficient algorithms.
Paper Structure (28 sections, 24 theorems, 222 equations, 2 figures)

This paper contains 28 sections, 24 theorems, 222 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Main Result: MMSE discontinuities imply Stable Algorithm Failure
  4. Applications
  5. The Planted Shortest Path Model
  6. Random Linear Code
  7. Results for Gaussian Subset Sum (GSS)
  8. Getting started: Preliminaries for the Proofs
  9. Proof of the key result: MMSE instability implies stable algorithm failure
  10. Proofs for Planted Shortest Path
  11. Noisy MMSE lower bound: Proof of Theorem \ref{['thm:noisy-MMSE-PSP']}
  12. Stability of Symmetric Low-Degree Polynomials: Proof of Theorem \ref{['PSP:stab']}
  13. Proofs for Random Linear Code
  14. Noisy MMSE: Proof of Theorem \ref{['thm:linear-code-noisy-MMSE']}
  15. Stability of Low-Degree Polynomials: Proof of Theorem \ref{['thm:low-deg-stability-rlc']}
  16. ...and 13 more sections

Key Result

Theorem 1.2

Suppose that for some $\rho\in[0,1]$ and $\alpha>0$, the MMSE is $(\rho,\alpha)$-unstable in the sense, Then any $(\rho,o(\alpha^2))$-stable algorithm $\mathcal{A}$ is $\Omega(\alpha)$-suboptimal:

Figures (2)

  • Figure 1: A pictorial representation of a sharp MMSE jump under small noise, which leads to stable algorithm failure per Theorem \ref{['thm:informal_main']}.
  • Figure 2: Pictorial representation of our separation results between the class polynomial-time algorithms can solve, and the classes stable algorithms/low-degree polynomials can solve for parametric estimation.

Theorems & Definitions (67)

  • Definition 1.1: Stable algorithm
  • Theorem 1.2: Informal: MMSE jumps imply stable algorithm failure
  • Theorem 2.1
  • Corollary 2.2
  • proof
  • Definition 3.1
  • Remark 3.2
  • Remark 3.3
  • Remark 3.4: "Polynomial-time solvability of PSP"
  • Definition 3.5: Noise operator for PSP
  • ...and 57 more