Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hardness of High-Dimensional Linear Classification

Alexander Munteanu, Simon Omlor, Jeff M. Phillips

Abstract

We establish new exponential in dimension lower bounds for the Maximum Halfspace Discrepancy problem, which models linear classification. Both are fundamental problems in computational geometry and machine learning in their exact and approximate forms. However, only $O(n^d)$ and respectively $\tilde O(1/\varepsilon^d)$ upper bounds are known and complemented by polynomial lower bounds that do not support the exponential in dimension dependence. We close this gap up to polylogarithmic terms by reduction from widely-believed hardness conjectures for Affine Degeneracy testing and $k$-Sum problems. Our reductions yield matching lower bounds of $\tildeΩ(n^d)$ and respectively $\tildeΩ(1/\varepsilon^d)$ based on Affine Degeneracy testing, and $\tildeΩ(n^{d/2})$ and respectively $\tildeΩ(1/\varepsilon^{d/2})$ conditioned on $k$-Sum. The first bound also holds unconditionally if the computational model is restricted to make sidedness queries, which corresponds to a widely spread setting implemented and optimized in many contemporary algorithms and computing paradigms.

Hardness of High-Dimensional Linear Classification

Abstract

We establish new exponential in dimension lower bounds for the Maximum Halfspace Discrepancy problem, which models linear classification. Both are fundamental problems in computational geometry and machine learning in their exact and approximate forms. However, only and respectively upper bounds are known and complemented by polynomial lower bounds that do not support the exponential in dimension dependence. We close this gap up to polylogarithmic terms by reduction from widely-believed hardness conjectures for Affine Degeneracy testing and -Sum problems. Our reductions yield matching lower bounds of and respectively based on Affine Degeneracy testing, and and respectively conditioned on -Sum. The first bound also holds unconditionally if the computational model is restricted to make sidedness queries, which corresponds to a widely spread setting implemented and optimized in many contemporary algorithms and computing paradigms.
Paper Structure (18 sections, 6 theorems, 3 equations)

This paper contains 18 sections, 6 theorems, 3 equations.

Table of Contents

  1. Introduction
  2. Linear Classification.
  3. Conditional Hardness.
  4. Our Results
  5. k-Sum Hardness.
  6. Affine Degeneracy Testing Hardness.
  7. Technical Overview
  8. Implications Beyond Machine Learning
  9. Additional Related Work
  10. Maximum Discrepancy Classification.
  11. More Connections to Learning Theory.
  12. No-Dimensional Results in Computational Geometry and Classification.
  13. Impossibility of No-Dimensional Results.
  14. Improved Halfspace Discrepancy Lower Bounds
  15. k-Sum Hardness
  16. ...and 3 more sections

Key Result

Theorem 3

Assume that there exists no algorithm that solves $k$- Sum on $n$ items within runtime $O(n^{\lceil k/2 \rceil - c})$ for any $c > 0$. Then there exist sets $R,B \subset \mathbb{R}^d, d=k-1$, each of size $n/2$, for which there is no algorithm that solves either

Theorems & Definitions (8)

  • Conjecture 1: $k$-Sum conjecture
  • Conjecture 2: Affine Degeneracy conjecture
  • Theorem 3: informal version of \ref{['thm:ksum']}
  • Theorem 4: informal version of \ref{['thm:affinedegeneracy']}
  • Theorem 5
  • Theorem 6
  • Lemma 7
  • Theorem 8