Iterative thresholding for non-linear learning in the strong $\varepsilon$-contamination model

TL;DR

The paper develops gradient-descent based iterative thresholding algorithms for robustly learning single-neuron models under the strong $ε$-contamination model, with corruption in both labels and covariates. It provides explicit approximation bounds for nonlinear activations (sigmoid, leaky-ReLU, ReLU) and linear regression, showing near-optimal dependence on contamination level $ε$ and noise variance $ν$, while achieving favorable sample complexities and runtime improvements. For nonlinear neurons, the main result is an $O(ν\sqrt{ε\log(1/ε)})$-type error with sample complexity $O(d/ε)$ and failure probability $e^{-Ω(d)}$; for linear regression the bound tightens to $O(νε\log(1/ε))$, with significant runtime reductions over prior work. The methods directly handle corrupted covariates using only spectral properties of the (uncorrupted) covariance, yielding practical robustness and broad applicability across activation functions. The work advances theory on robust learning with iterative thresholding beyond GLMs and sets the stage for extensions to broader neural architectures.

Abstract

We derive approximation bounds for learning single neuron models using thresholded gradient descent when both the labels and the covariates are possibly corrupted adversarially. We assume the data follows the model $y = σ(\mathbf{w}^{*} \cdot \mathbf{x}) + ξ,$ where $σ$ is a nonlinear activation function, the noise $ξ$ is Gaussian, and the covariate vector $\mathbf{x}$ is sampled from a sub-Gaussian distribution. We study sigmoidal, leaky-ReLU, and ReLU activation functions and derive a $O(ν\sqrt{ε\log(1/ε)})$ approximation bound in $\ell_{2}$-norm, with sample complexity $O(d/ε)$ and failure probability $e^{-Ω(d)}$. We also study the linear regression problem, where $σ(\mathbf{x}) = \mathbf{x}$. We derive a $O(νε\log(1/ε))$ approximation bound, improving upon the previous $O(ν)$ approximation bounds for the gradient-descent based iterative thresholding algorithms of Bhatia et al. (NeurIPS 2015) and Shen and Sanghavi (ICML 2019). Our algorithm has a $O(\textrm{polylog}(N,d)\log(R/ε))$ runtime complexity when $\|\mathbf{w}^{*}\|_2 \leq R$, improving upon the $O(\text{polylog}(N,d)/ε^2)$ runtime complexity of Awasthi et al. (NeurIPS 2022).

Theorems & Definitions (35)

• Definition 1: Strong $\epsilon$-Contamination Model
• Theorem 2: Theorem 5 in bhatia2015robust
• Theorem 3: Theorem 4.2 in awasthi:2022
• Definition 4: Sub-Gaussian Distribution
• Definition 5
• Definition 6: Hard Thresholding Operator
• Theorem 7
• Corollary 8
• Theorem 12
• Theorem 13