Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Linear Separability under Linear Compression with Applications to Hard Support Vector Machine

Paul McVay, Tie Liu, Krishna Narayanan

TL;DR

The paper asks when linear separability of a data-generating distribution is preserved under linear compression. It develops a margin-geometry relationship using the hard-SVM framework and introduces the notion of $\\eta$-inner-product preservation and $w^*$-compatibility to extend finite-sample ideas to infinite data supports. The main result shows that separability is preserved after compression if the inner-product distortion satisfies $\\eta<1/\\|w_0\\|^2$, and it provides concrete bounds on compression length for sub-Gaussian matrices as well as a generalization bound for compressed hard-SVM. These findings yield distribution-level guarantees for compressed learning with SVMs and inform practical compression lengths and expected generalization performance.

Abstract

This paper investigates the theoretical problem of maintaining linear separability of the data-generating distribution under linear compression. While it has been long known that linear separability may be maintained by linear transformations that approximately preserve the inner products between the domain points, the limit to which the inner products are preserved in order to maintain linear separability was unknown. In this paper, we show that linear separability is maintained as long as the distortion of the inner products is smaller than the squared margin of the original data-generating distribution. The proof is mainly based on the geometry of hard support vector machines (SVM) extended from the finite set of training examples to the (possibly) infinite domain of the data-generating distribution. As applications, we derive bounds on the (i) compression length of random sub-Gaussian matrices; and (ii) generalization error for compressive learning with hard-SVM.

On Linear Separability under Linear Compression with Applications to Hard Support Vector Machine

TL;DR

The paper asks when linear separability of a data-generating distribution is preserved under linear compression. It develops a margin-geometry relationship using the hard-SVM framework and introduces the notion of -inner-product preservation and -compatibility to extend finite-sample ideas to infinite data supports. The main result shows that separability is preserved after compression if the inner-product distortion satisfies , and it provides concrete bounds on compression length for sub-Gaussian matrices as well as a generalization bound for compressed hard-SVM. These findings yield distribution-level guarantees for compressed learning with SVMs and inform practical compression lengths and expected generalization performance.

Abstract

This paper investigates the theoretical problem of maintaining linear separability of the data-generating distribution under linear compression. While it has been long known that linear separability may be maintained by linear transformations that approximately preserve the inner products between the domain points, the limit to which the inner products are preserved in order to maintain linear separability was unknown. In this paper, we show that linear separability is maintained as long as the distortion of the inner products is smaller than the squared margin of the original data-generating distribution. The proof is mainly based on the geometry of hard support vector machines (SVM) extended from the finite set of training examples to the (possibly) infinite domain of the data-generating distribution. As applications, we derive bounds on the (i) compression length of random sub-Gaussian matrices; and (ii) generalization error for compressive learning with hard-SVM.
Paper Structure (12 sections, 15 theorems, 63 equations)

This paper contains 12 sections, 15 theorems, 63 equations.

Table of Contents

  1. Introduction
  2. Problem setup and main results
  3. Bounds on the compression length of random sub-Gaussian matrices
  4. Generalization bound for compressed learning with hard-SVM
  5. Proofs
  6. Proof of Proposition \ref{['prop:comp']}
  7. Proof of Theorem \ref{['thm:main']}
  8. Appendix
  9. Proof of Proposition \ref{['prop:sparse']}
  10. Proof of Lemma \ref{['lm:support']}
  11. Proof of Lemma \ref{['lm:strict-separation']}
  12. Proof of Lemma \ref{['lm:achieve-min-sep']}

Key Result

Theorem 1

Assume that the data-generating distribution $\mu$ is linearly separable by the hyperplane in $\mathbb{R}^n$ indexed by $(w_0,b_0)$. Let $Q\in\mathbb{R}^{m\times n}$ be $\eta$-inner-product-preserving over $\mathcal{X}$, where $\mathcal{X}$ is the support of $\mu_{\mathsf{x}}$. Then, $\mu$ remains l

Theorems & Definitions (18)

  • Definition 1: Inner product preserving linear transformation
  • Theorem 1
  • Definition 2: Compatible vectors
  • Proposition 1: Existence of a compatible solution
  • Definition 3: squared distance preserving linear transform
  • Proposition 2
  • Proposition 3
  • Theorem 2: foucart2013mathematical
  • Theorem 3
  • Theorem 4: vershynin2018high
  • ...and 8 more