Robust learning of halfspaces under log-concave marginals
Jane Lange, Arsen Vasilyan
TL;DR
This work addresses robust learning of halfspaces under isotropic subgaussian log-concave marginals by introducing RobustLearn, an improper agnostic learner that achieves both near-optimal classification error and adversarial robustness. The method combines three components: LearnRealValued, a convex program that finds a low-noise-sensitivity polynomial p with err ≤ opt + O(ε) and NS ≤ O(r+ε); ComputeClassifier, a deterministic four-piece rounding strategy derived via Carathéodory’s theorem to bound error, boundary volume, and isolation; and RobustnessLCA, a local correction that smooths to a globally adversarially robust classifier. The algorithm runs in time d^{ ilde{O}(1/ε^2)} with sample complexity of the same order, matching the efficiency of the best-known improper agnostic learners and extending robustness guarantees beyond Gaussian to log-concave marginals. Additionally, the work provides a verifiable robustness framework under standard complexity assumptions, enabling efficient certification of robustness for the learned model. Overall, it delivers a theoretically principled, computationally feasible path to robust halfspace learning in high dimensions with verifiable robustness properties.
Abstract
We say that a classifier is \emph{adversarially robust} to perturbations of norm $r$ if, with high probability over a point $x$ drawn from the input distribution, there is no point within distance $\le r$ from $x$ that is classified differently. The \emph{boundary volume} is the probability that a point falls within distance $r$ of a point with a different label. This work studies the task of computationally efficient learning of hypotheses with small boundary volume, where the input is distributed as a subgaussian isotropic log-concave distribution over $\mathbb{R}^d$. Linear threshold functions are adversarially robust; they have boundary volume proportional to $r$. Such concept classes are efficiently learnable by polynomial regression, which produces a polynomial threshold function (PTF), but PTFs in general may have boundary volume $Ω(1)$, even for $r \ll 1$. We give an algorithm that agnostically learns linear threshold functions and returns a classifier with boundary volume $O(r+\varepsilon)$ at radius of perturbation $r$. The time and sample complexity of $d^{\tilde{O}(1/\varepsilon^2)}$ matches the complexity of polynomial regression. Our algorithm augments the classic approach of polynomial regression with three additional steps: a) performing the $\ell_1$-error regression under noise sensitivity constraints, b) a structured partitioning and rounding step that returns a Boolean classifier with error $\textsf{opt} + O(\varepsilon)$ and noise sensitivity $O(r+\varepsilon)$ simultaneously, and c) a local corrector that ``smooths'' a function with low noise sensitivity into a function that is adversarially robust.