Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models

Leonardo Defilippis, Florent Krzakala, Bruno Loureiro, Antoine Maillard

Abstract

Understanding when learning is statistically possible yet computationally hard is a central challenge in high-dimensional statistics. In this work, we investigate this question in the context of single- and multi-index models, classes of functions widely studied as benchmarks to probe the ability of machine learning methods to discover features in high-dimensional data. Our main contribution is to show that a Noise Sensitivity Exponent (NSE) - a simple quantity determined by the activation function - governs the existence and magnitude of statistical-to-computational gaps within a broad regime of these models. We first establish that, in single-index models with large additive noise, the onset of a computational bottleneck is fully characterized by the NSE. We then demonstrate that the same exponent controls a statistical-computational gap in the specialization transition of large separable multi-index models, where individual components become learnable. Finally, in hierarchical multi-index models, we show that the NSE governs the optimal computational rate in which different directions are sequentially learned. Taken together, our results identify the NSE as a unifying property linking noise robustness, computational hardness, and feature specialization in high-dimensional learning.

A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models

Abstract

Understanding when learning is statistically possible yet computationally hard is a central challenge in high-dimensional statistics. In this work, we investigate this question in the context of single- and multi-index models, classes of functions widely studied as benchmarks to probe the ability of machine learning methods to discover features in high-dimensional data. Our main contribution is to show that a Noise Sensitivity Exponent (NSE) - a simple quantity determined by the activation function - governs the existence and magnitude of statistical-to-computational gaps within a broad regime of these models. We first establish that, in single-index models with large additive noise, the onset of a computational bottleneck is fully characterized by the NSE. We then demonstrate that the same exponent controls a statistical-computational gap in the specialization transition of large separable multi-index models, where individual components become learnable. Finally, in hierarchical multi-index models, we show that the NSE governs the optimal computational rate in which different directions are sequentially learned. Taken together, our results identify the NSE as a unifying property linking noise robustness, computational hardness, and feature specialization in high-dimensional learning.
Paper Structure (33 sections, 7 theorems, 154 equations, 1 figure)

This paper contains 33 sections, 7 theorems, 154 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Setting, definitions, and related works
  3. Target function
  4. High-dimensional limit ---
  5. Concrete cases ---
  6. The Noise Sensitivity Exponent (NSE)
  7. Examples, and related exponents
  8. Further related works
  9. Statistical-computational gap in even single-index models at high noise
  10. Computational weak recovery ---
  11. Information-theoretic weak recovery ---
  12. The specialization transition in large separable multi-index models
  13. Scaling laws in hierarchical multi-index models
  14. Sketch of proofs and derivations
  15. Single-index models at large noise
  16. ...and 18 more sections

Key Result

Theorem 3.2

Consider the single-index model of eq. eq:sindex_model. Assume that the even function $\sigma$ satisfies the assumptions of Definition def:nse, and has NSE $\beta_\star < \infty$. Then, the optimal AMP algorithm achieves weak recovery (see Definition def:wrecovery) exactly for $\alpha > \alpha^{\rm

Figures (1)

  • Figure 1: Examples of computational weak recovery thresholds $\alpha^{\rm Alg.}_{\rm WR}$mondelli2018fundamentalBarbier2019 as a function of the signal-to-noise ratio $\lambda$, for the single-index model of eq. \ref{['eq:sindex_model']}.

Theorems & Definitions (12)

  • Definition 2.1: Noise Sensitivity Exponent
  • Definition 3.1: Weak recovery
  • Theorem 3.2
  • Theorem 3.3
  • Definition 4.1: Weak recovery and Specialization
  • Theorem 4.2
  • Remark 4.4
  • Definition 5.1: Weak recovery of the $k^{\rm th}$ feature.
  • Theorem 5.2
  • Corollary 5.3
  • ...and 2 more