Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SoS Certificates for Sparse Singular Values and Their Applications: Robust Statistics, Subspace Distortion, and More

Ilias Diakonikolas, Samuel B. Hopkins, Ankit Pensia, Stefan Tiegel

TL;DR

The paper tackles the problem of certifying sparse singular values for Gaussian and subgaussian random matrices, introducing a Sum-of-Squares framework that yields certificates beating naive spectral bounds in broad regimes. The core method converts operator-norm bounds into Schatten-$p$ norm certificates via graph-polynomial representations and the Efron–Stein decomposition, enabling tight spectral control of complex polynomial forms. These certificates drive algorithmic advances across robust statistics, subspace distortion, and 2-to-$p$ norm tasks, achieving near-optimal tradeoffs between sample complexity, contamination, and computation, with subgaussian extensions. The work also establishes low-degree hardness results that align with the proposed regimes, outlining fundamental computational limits and connections to private estimation and NGCA. Overall, the results provide a unified, SOS-based toolkit for probabilistic matrix problems with sparse structure, with wide implications for robust learning and high-dimensional certification.

Abstract

We study $\textit{sparse singular value certificates}$ for random rectangular matrices. If $M$ is an $n \times d$ matrix with independent Gaussian entries, we give a new family of polynomial-time algorithms which can certify upper bounds on the maximum of $\|M u\|$, where $u$ is a unit vector with at most $ηn$ nonzero entries for a given $η\in (0,1)$. This basic algorithmic primitive lies at the heart of a wide range of problems across algorithmic statistics and theoretical computer science. Our algorithms certify a bound which is asymptotically smaller than the naive one, given by the maximum singular value of $M$, for nearly the widest-possible range of $n,d,$ and $η$. Efficiently certifying such a bound for a range of $n,d$ and $η$ which is larger by any polynomial factor than what is achieved by our algorithm would violate lower bounds in the SQ and low-degree polynomials models. Our certification algorithm makes essential use of the Sum-of-Squares hierarchy. To prove the correctness of our algorithm, we develop a new combinatorial connection between the graph matrix approach to analyze random matrices with dependent entries, and the Efron-Stein decomposition of functions of independent random variables. As applications of our certification algorithm, we obtain new efficient algorithms for a wide range of well-studied algorithmic tasks. In algorithmic robust statistics, we obtain new algorithms for robust mean and covariance estimation with tradeoffs between breakdown point and sample complexity, which are nearly matched by SQ and low-degree polynomial lower bounds (that we establish). We also obtain new polynomial-time guarantees for certification of $\ell_1/\ell_2$ distortion of random subspaces of $\mathbb{R}^n$ (also with nearly matching lower bounds), sparse principal component analysis, and certification of the $2\rightarrow p$ norm of a random matrix.

SoS Certificates for Sparse Singular Values and Their Applications: Robust Statistics, Subspace Distortion, and More

TL;DR

The paper tackles the problem of certifying sparse singular values for Gaussian and subgaussian random matrices, introducing a Sum-of-Squares framework that yields certificates beating naive spectral bounds in broad regimes. The core method converts operator-norm bounds into Schatten- norm certificates via graph-polynomial representations and the Efron–Stein decomposition, enabling tight spectral control of complex polynomial forms. These certificates drive algorithmic advances across robust statistics, subspace distortion, and 2-to- norm tasks, achieving near-optimal tradeoffs between sample complexity, contamination, and computation, with subgaussian extensions. The work also establishes low-degree hardness results that align with the proposed regimes, outlining fundamental computational limits and connections to private estimation and NGCA. Overall, the results provide a unified, SOS-based toolkit for probabilistic matrix problems with sparse structure, with wide implications for robust learning and high-dimensional certification.

Abstract

We study for random rectangular matrices. If is an matrix with independent Gaussian entries, we give a new family of polynomial-time algorithms which can certify upper bounds on the maximum of , where is a unit vector with at most nonzero entries for a given . This basic algorithmic primitive lies at the heart of a wide range of problems across algorithmic statistics and theoretical computer science. Our algorithms certify a bound which is asymptotically smaller than the naive one, given by the maximum singular value of , for nearly the widest-possible range of and . Efficiently certifying such a bound for a range of and which is larger by any polynomial factor than what is achieved by our algorithm would violate lower bounds in the SQ and low-degree polynomials models. Our certification algorithm makes essential use of the Sum-of-Squares hierarchy. To prove the correctness of our algorithm, we develop a new combinatorial connection between the graph matrix approach to analyze random matrices with dependent entries, and the Efron-Stein decomposition of functions of independent random variables. As applications of our certification algorithm, we obtain new efficient algorithms for a wide range of well-studied algorithmic tasks. In algorithmic robust statistics, we obtain new algorithms for robust mean and covariance estimation with tradeoffs between breakdown point and sample complexity, which are nearly matched by SQ and low-degree polynomial lower bounds (that we establish). We also obtain new polynomial-time guarantees for certification of distortion of random subspaces of (also with nearly matching lower bounds), sparse principal component analysis, and certification of the norm of a random matrix.
Paper Structure (106 sections, 44 theorems, 220 equations, 4 figures, 3 algorithms)

This paper contains 106 sections, 44 theorems, 220 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Brief overview of results.
  3. Main Algorithmic Result: Sparse Singular Value Certificates
  4. Summary of prior work.
  5. Main algorithmic contribution.
  6. Almost matching lower bounds.
  7. Extensions of \ref{['thm:main-intro']}.
  8. Algorithmic Applications of Main Result
  9. Algorithmic Robust Statistics
  10. Discussion on contamination rate.
  11. Robust covariance estimation.
  12. Robust covariance-aware mean estimation.
  13. Robust mean estimation in Euclidean norm.
  14. Subspace Distortion and Planted Sparse Vectors
  15. Consequences for planted sparse vector.
  16. ...and 91 more sections

Key Result

Theorem 1.5

For every $\epsilon > 0$, there is an $(nd)^{O(1/\epsilon)}$-time algorithm which certifies that the maximum $\eta$-sparse singular value of $\tfrac{1}{\sqrt n} M$ is $o(1)$ if $n \gg \eta^2 \, d^{2+\epsilon}$ and $\eta \leq o(1)$. In particular, choosing $\epsilon = 1/\log(d)$, there is an algorith

Figures (4)

  • Figure 1: Minimum value of $\tfrac{n}{d^2}$ as a function of $\eta$ for polynomial-time $\eta$-SSV refutation algorithms. Note the discontinuity in the state of the art guarantees preceding our work.
  • Figure 2: Illustration of vertex merging process.
  • Figure 3: Example of a sequence of circle and diamond merges and edge removals to obtain an admissible graph matrix.
  • Figure 4: An illustration of \ref{['lem:union-of-cycles-merge']}.

Theorems & Definitions (119)

  • Definition 1.2: $\eta$-Sparse Singular Value Certification
  • Theorem 1.5: Main result; see \ref{['thm:sparse_sing_val_full']} for the full version
  • Definition 1.6: Strong Contamination Model DiaKKLMS16-focs
  • Theorem 1.7: Robust Covariance Estimation
  • Theorem 1.8: Robust Covariance-Aware Mean Estimation
  • Theorem 1.9: Robust Mean Estimation
  • Remark 1.10: Connections to Private Mean Estimation
  • Definition 1.11: Distortion of a Subspace
  • Theorem 1.13: This work; see \ref{['thm:distortion_full']} for the full version
  • Definition 3.1
  • ...and 109 more