Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fully Polynomial-Time Algorithm for Robustly Learning Halfspaces over the Hypercube

Gautam Chandrasekaran, Adam R. Klivans, Konstantinos Stavropoulos, Arsen Vasilyan

TL;DR

This work delivers the first fully polynomial-time algorithm for agnostically learning halfspaces under the uniform distribution on the Boolean hypercube in the presence of contamination, achieving error on the order of ${ m opt}^{O(1)}+ε$. The authors blend a robust phase-based approach with Generalized Linear Models, leveraging a heavy-coefficient GLM component, hinge-loss learning for regular tail coefficients, and a robust identification of influential coordinates via Chow-parameter-inspired techniques. A key technical advance is a GLM-learning procedure with runtime polylogarithmic in the activation’s Lipschitz constant, enabling efficient learning of sigmoidal activations with weights bounded independently of dimension. The framework tolerates bounded contamination and, via a reduction to adaptive vs. oblivious adversaries, yields poly(d,1/ε)-time guarantees with error that scales as a function of the contamination rate. Overall, the paper shows that supervised learning with discrete distributions such as the hypercube can be robustly and efficiently learned, bridging a notable gap between discrete and continuous distribution settings and extending GLM tools to robust, high-dimensional, discrete learning problems.

Abstract

We give the first fully polynomial-time algorithm for learning halfspaces with respect to the uniform distribution on the hypercube in the presence of contamination, where an adversary may corrupt some fraction of examples and labels arbitrarily. We achieve an error guarantee of $η^{O(1)}+ε$ where $η$ is the noise rate. Such a result was not known even in the agnostic setting, where only labels can be adversarially corrupted. All prior work over the last two decades has a superpolynomial dependence in $1/ε$ or succeeds only with respect to continuous marginals (such as log-concave densities). Previous analyses rely heavily on various structural properties of continuous distributions such as anti-concentration. Our approach avoids these requirements and makes use of a new algorithm for learning Generalized Linear Models (GLMs) with only a polylogarithmic dependence on the activation function's Lipschitz constant. More generally, our framework shows that supervised learning with respect to discrete distributions is not as difficult as previously thought.

A Fully Polynomial-Time Algorithm for Robustly Learning Halfspaces over the Hypercube

TL;DR

This work delivers the first fully polynomial-time algorithm for agnostically learning halfspaces under the uniform distribution on the Boolean hypercube in the presence of contamination, achieving error on the order of . The authors blend a robust phase-based approach with Generalized Linear Models, leveraging a heavy-coefficient GLM component, hinge-loss learning for regular tail coefficients, and a robust identification of influential coordinates via Chow-parameter-inspired techniques. A key technical advance is a GLM-learning procedure with runtime polylogarithmic in the activation’s Lipschitz constant, enabling efficient learning of sigmoidal activations with weights bounded independently of dimension. The framework tolerates bounded contamination and, via a reduction to adaptive vs. oblivious adversaries, yields poly(d,1/ε)-time guarantees with error that scales as a function of the contamination rate. Overall, the paper shows that supervised learning with discrete distributions such as the hypercube can be robustly and efficiently learned, bridging a notable gap between discrete and continuous distribution settings and extending GLM tools to robust, high-dimensional, discrete learning problems.

Abstract

We give the first fully polynomial-time algorithm for learning halfspaces with respect to the uniform distribution on the hypercube in the presence of contamination, where an adversary may corrupt some fraction of examples and labels arbitrarily. We achieve an error guarantee of where is the noise rate. Such a result was not known even in the agnostic setting, where only labels can be adversarially corrupted. All prior work over the last two decades has a superpolynomial dependence in or succeeds only with respect to continuous marginals (such as log-concave densities). Previous analyses rely heavily on various structural properties of continuous distributions such as anti-concentration. Our approach avoids these requirements and makes use of a new algorithm for learning Generalized Linear Models (GLMs) with only a polylogarithmic dependence on the activation function's Lipschitz constant. More generally, our framework shows that supervised learning with respect to discrete distributions is not as difficult as previously thought.

Paper Structure

This paper contains 32 sections, 19 theorems, 161 equations, 1 figure, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Polynomial-Time Robust Learning Algorithms under Continuous Distributions.
  3. Robust Learning of Halfspaces over the Hypercube.
  4. Learning Generalized Linear Models.
  5. Brief Technical Overview
  6. Algorithm Outline.
  7. Proof Outline.
  8. Structured Case.
  9. Sparse Case.
  10. Learning the Heavy Coefficients: A Connection with GLM Learning
  11. Learning the Regular Coefficients: Hinge Loss Minimization
  12. Finding the Positions of Influential Variables
  13. Tolerating Contamination
  14. Equivalence Between Adaptive and Oblivious Adversaries.
  15. Finding Positions of Influential Variables under Contamination.
  16. ...and 17 more sections

Key Result

Theorem 1

There is an algorithm that, given sample access to a distribution over pairs $(\bm{x},y)\in \mathbb{R}^d\times \{\pm 1\}$ where $\|\bm{x}\|_2 \le \sqrt{d}$ and $\mathop{\mathrm{\mathbb{E}}}\limits[y | \bm{x}] = \sigma^{\mathrm{sig}}(\bm{w}^* \cdot \bm{x} + \tau^*)$ for some unknown $\bm{w}^*,\tau^*$ The sample complexity of the algorithm is $\tilde{O}(d / \varepsilon^2)$ and its time complexity is

Figures (1)

  • Figure 1: Reduction tree for the proof of our main result (\ref{['thm: main theorem']}). Each internal node corresponds to a learning problem that reduces to solving all of its children. The dashed arrow indicates a dependency between two leaf nodes, where the output of one serves as input to the other.

Theorems & Definitions (64)

  • Theorem 1
  • Theorem 2
  • Theorem 3: Simplified version of \ref{['theorem:influential']}
  • Definition 4: Learning with Bounded Contamination BSHOUTY2002255
  • Theorem 5
  • Definition 6
  • Definition 7
  • Definition 8
  • Lemma 9
  • Claim 10
  • ...and 54 more