Minimax learning rates for estimating binary classifiers under margin conditions

TL;DR

The paper addresses minimax rates for estimating noiseless binary classifiers whose decision boundaries are horizon functions under a geometric margin condition with exponent $\gamma>0$. It develops a general minimax framework based on entropy numbers $V$ and $M$, then specializes to three function classes—Barron regular, Hölder, and convex—to derive explicit rates. Notably, it shows near-fast rates $\mathcal{I}(\mathcal{B}_C) \approx n^{-1}$ (up to logs) for Barron functions under strong margins, and rates $\mathcal{I}(\mathcal{H}_{\alpha}) \approx n^{-\frac{\gamma\alpha}{\gamma\alpha+d_1}}$ for Hölder classes, with margin-driven improvements that can overcome the curse of dimensionality. For convex boundary classes, rates depend on the margin and interpolation parameters, highlighting how margin conditions shape fundamental learning limits in noiseless classification.

Abstract

We study classification problems using binary estimators where the decision boundary is described by horizon functions and where the data distribution satisfies a geometric margin condition. We establish upper and lower bounds for the minimax learning rate over broad function classes with bounded Kolmogorov entropy in Lebesgue norms. A key novelty of our work is the derivation of lower bounds on the worst-case learning rates under a geometric margin condition -- a setting that is almost universally satisfied in practice but remains theoretically challenging. Moreover, our results deal with the noiseless setting, where lower bounds are particularly hard to establish. We apply our general results to classification problems with decision boundaries belonging to several function classes: for Barron-regular functions, and for Hölder-continuous functions with strong margins, we identify optimal rates close to the fast learning rates of $\mathcal{O}(n^{-1})$ for $n \in \mathbb{N}$ samples. Also for merely convex decision boundaries, in a strong margin case optimal rates near $\mathcal{O}(n^{-1/2})$ can be achieved.

Figures (1)

• Figure 1: Geometric margin in common classification problems. The top row shows a two dimensional embedding on the first two principle components and a decision boundary identified by a support vector machine. Clearly MNIST MNISTref and Fashion MNIST FashionMNISTref exhibit a strong margin between some classes. For the CIFAR-10 data CIFAR10ref the margin is not visible in the two dimensional embedding. In the second row, we show the class probabilities predicted by a support vector classifier, which again shows extremely strong margin for MNIST and Fashon MNIST, but also reveals that the CIFAR-10 data exhibits a margin, albeit a weaker one. Which lower bounds on learning can be found in the presence of such various types of margins will be demonstrated in our main results Theorem \ref{['mainteo']} and Corollary \ref{['maincoro']}.

Theorems & Definitions (12)

• Remark 1
• Lemma 2
• Lemma 3
• Lemma 4
• Theorem 5
• Remark 6
• Corollary 7
• Remark 8
• Lemma 9
• Remark 10