Super-fast rates of convergence for Neural Networks Classifiers under the Hard Margin Condition

TL;DR

The paper addresses binary classification with deep ReLU networks under Tsybakov's margin framework, including the hard-margin limit, and proves non-asymptotic excess-risk bounds. By introducing a novel excess-risk decomposition and leveraging approximation theory for smooth regression functions, it shows that ARMs with square-loss surrogates and $\ell_p$ penalties can achieve arbitrarily fast rates $\mathcal{O}(n^{-\alpha})$ with suitable network architectures, and exponential rates in a teacher-student setup. Under the hard-margin condition, the exponent can scale with the smoothness $s$, yielding super-fast convergence, while the well-specified teacher-student case achieves exponential decay regardless of network depth. The results provide a rigorous theory explaining how deep networks can attain CoD-free, even super-fast, learning rates in separable-margin regimes, with explicit, non-asymptotic bounds and constructive architecture guidance.

Abstract

We study the classical binary classification problem for hypothesis spaces of Deep Neural Networks (DNNs) with ReLU activation under Tsybakov's low-noise condition with exponent $q>0$, and its limit-case $q\to\infty$ which we refer to as the "hard-margin condition". We show that DNNs which minimize the empirical risk with square loss surrogate and $\ell_p$ penalty can achieve finite-sample excess risk bounds of order $\mathcal{O}\left(n^{-α}\right)$ for arbitrarily large $α>0$ under the hard-margin condition, provided that the regression function $η$ is sufficiently smooth. The proof relies on a novel decomposition of the excess risk which might be of independent interest.

Theorems & Definitions (26)

• Lemma 1
• proof
• Lemma 2
• proof : Proof of Lemma \ref{['lemma:finite_sup']}
• Definition 1: $\varepsilon$-cover
• Definition 2: $\varepsilon$-covering number
• Definition 3
• Theorem 1
• Lemma 3: Theorem 2.6 in berner2020analysis
• Lemma 4