Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Outlier-robust Mean Estimation near the Breakdown Point via Sum-of-Squares

Hongjie Chen, Deepak Narayanan Sridharan, David Steurer

TL;DR

A new analysis of the canonical sum-of-squares program introduced in kothari2018robust is given and it is shown that this program efficiently achieves optimal error rate for all $\varepsilon \in[0,\frac{1}{2})$.

Abstract

We revisit the problem of estimating the mean of a high-dimensional distribution in the presence of an $\varepsilon$-fraction of adversarial outliers. When $\varepsilon$ is at most some sufficiently small constant, previous works can achieve optimal error rate efficiently \cite{diakonikolas2018robustly, kothari2018robust}. As $\varepsilon$ approaches the breakdown point $\frac{1}{2}$, all previous algorithms incur either sub-optimal error rates or exponential running time. In this paper we give a new analysis of the canonical sum-of-squares program introduced in \cite{kothari2018robust} and show that this program efficiently achieves optimal error rate for all $\varepsilon \in[0,\frac{1}{2})$. The key ingredient for our results is a new identifiability proof for robust mean estimation that focuses on the overlap between the distributions instead of their statistical distance as in previous works. We capture this proof within the sum-of-squares proof system, thus obtaining efficient algorithms using the sum-of-squares proofs to algorithms paradigm \cite{raghavendra2018high}.

Outlier-robust Mean Estimation near the Breakdown Point via Sum-of-Squares

TL;DR

A new analysis of the canonical sum-of-squares program introduced in kothari2018robust is given and it is shown that this program efficiently achieves optimal error rate for all .

Abstract

We revisit the problem of estimating the mean of a high-dimensional distribution in the presence of an -fraction of adversarial outliers. When is at most some sufficiently small constant, previous works can achieve optimal error rate efficiently \cite{diakonikolas2018robustly, kothari2018robust}. As approaches the breakdown point , all previous algorithms incur either sub-optimal error rates or exponential running time. In this paper we give a new analysis of the canonical sum-of-squares program introduced in \cite{kothari2018robust} and show that this program efficiently achieves optimal error rate for all . The key ingredient for our results is a new identifiability proof for robust mean estimation that focuses on the overlap between the distributions instead of their statistical distance as in previous works. We capture this proof within the sum-of-squares proof system, thus obtaining efficient algorithms using the sum-of-squares proofs to algorithms paradigm \cite{raghavendra2018high}.

Paper Structure

This paper contains 38 sections, 22 theorems, 114 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Results
  3. Bounded Covariance Distributions
  4. Certifiably bounded higher moment distributions
  5. Gaussian Distributions
  6. Technical Overview
  7. Notation
  8. Organization
  9. Robust Mean Estimation via Sum-of-Squares
  10. Optimal Error for Bounded Covariance
  11. Identifiability for large corruption
  12. SoSizing the proof of identifiability
  13. Robust Gaussian Mean Estimation with known Covariance
  14. Discussion about existing work
  15. Rounding
  16. ...and 23 more sections

Key Result

Theorem 1.2

Let $\mathcal{D}$ be a distribution on $\varmathbb R^d$ with mean $\mu^*$ and covariance $\Sigma^*$. Suppose $\Sigma^* \preccurlyeq \sigma^2\cdot I_d$. Let $\{ {x}^*_i \}_{i=1}^n \overset{\mathrm{iid}}{\sim} \mathcal{D}.$ Let $\{z_i\}_{i=1}^n$ be an $\varepsilon$-corruption of $\{ {x}^*_i \}_{i=1}^n with probability 0.99.

Theorems & Definitions (40)

  • Definition 1.1: $\varepsilon$-corruption
  • Theorem 1.2
  • Lemma 1.3: Lower Bound for Bounded Covariance
  • Definition 1.4: Distributions with Certifiably Bounded $k^{th}$ moments hopkins2018mixture
  • Theorem 1.5
  • Lemma 1.6: Lower Bound for Bounded Higher Moments
  • Theorem 1.7
  • Lemma 1.8: Lower Bound for Gaussians
  • Lemma 2.1
  • Theorem 2.2
  • ...and 30 more