Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Sparse Mean Estimation via Sum of Squares

Ilias Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, Thanasis Pittas

TL;DR

The first efficient algorithms for robust sparse mean estimation without a priori knowledge of the covariance are developed, providing evidence that the sample-time-error tradeoffs achieved by the algorithms are qualitatively the best possible.

Abstract

We study the problem of high-dimensional sparse mean estimation in the presence of an $ε$-fraction of adversarial outliers. Prior work obtained sample and computationally efficient algorithms for this task for identity-covariance subgaussian distributions. In this work, we develop the first efficient algorithms for robust sparse mean estimation without a priori knowledge of the covariance. For distributions on $\mathbb R^d$ with "certifiably bounded" $t$-th moments and sufficiently light tails, our algorithm achieves error of $O(ε^{1-1/t})$ with sample complexity $m = (k\log(d))^{O(t)}/ε^{2-2/t}$. For the special case of the Gaussian distribution, our algorithm achieves near-optimal error of $\tilde O(ε)$ with sample complexity $m = O(k^4 \mathrm{polylog}(d))/ε^2$. Our algorithms follow the Sum-of-Squares based, proofs to algorithms approach. We complement our upper bounds with Statistical Query and low-degree polynomial testing lower bounds, providing evidence that the sample-time-error tradeoffs achieved by our algorithms are qualitatively the best possible.

Robust Sparse Mean Estimation via Sum of Squares

TL;DR

The first efficient algorithms for robust sparse mean estimation without a priori knowledge of the covariance are developed, providing evidence that the sample-time-error tradeoffs achieved by the algorithms are qualitatively the best possible.

Abstract

We study the problem of high-dimensional sparse mean estimation in the presence of an -fraction of adversarial outliers. Prior work obtained sample and computationally efficient algorithms for this task for identity-covariance subgaussian distributions. In this work, we develop the first efficient algorithms for robust sparse mean estimation without a priori knowledge of the covariance. For distributions on with "certifiably bounded" -th moments and sufficiently light tails, our algorithm achieves error of with sample complexity . For the special case of the Gaussian distribution, our algorithm achieves near-optimal error of with sample complexity . Our algorithms follow the Sum-of-Squares based, proofs to algorithms approach. We complement our upper bounds with Statistical Query and low-degree polynomial testing lower bounds, providing evidence that the sample-time-error tradeoffs achieved by our algorithms are qualitatively the best possible.
Paper Structure (53 sections, 40 theorems, 120 equations, 4 algorithms)

This paper contains 53 sections, 40 theorems, 120 equations, 4 algorithms.

Table of Contents

  1. Introduction
  2. Our Results
  3. Overview of Techniques
  4. Robust Sparse Mean Estimation with Bounded Moments
  5. Identifiability in the Presence of Outliers
  6. Sampling Preserves Certifiably Bounded Moments
  7. Achieving Near-optimal Error for Gaussian Inliers
  8. Statistical Query and Low-Degree Testing Lower Bounds
  9. Prior and Related Work
  10. Organization
  11. Preliminaries
  12. Basic Notation
  13. Linear Algebra Notation
  14. Probability Notation
  15. SoS Preliminaries
  16. ...and 38 more sections

Key Result

Theorem 1.4

Let $t$ be a power of two, $D$ be a distribution on $\mathbb R^d$ with unknown mean $\mu$, and $\epsilon<\epsilon_0$ for a sufficiently small constant $\epsilon_0>0$. Suppose that $D$ has $t$-th moments certifiably bounded in $k$-sparse directions by $M$ (cf. def:bounded-moments-k-sparse) and subexp

Theorems & Definitions (104)

  • Definition 1.1: Strong Contamination Model
  • Definition 1.2: Certifiably $(M,t)$-Bounded Central Moments
  • Definition 1.3: (2, $k$)-norm
  • Theorem 1.4: Robust Sparse Mean Estimation for Certifiably Bounded Moments
  • Definition 1.4: STAT Oracle
  • Theorem 1.5: SQ Lower Bound for Subgaussian Distributions, Informal Statement
  • Theorem 1.6: Robust Sparse Gaussian Mean Estimation
  • Theorem 1.7: SQ Lower Bound for Gaussian Sparse Mean Estimation, Informal Statement
  • Definition 2.1: Bit complexity
  • Definition 2.2: Symbolic polynomial
  • ...and 94 more