Beyond Tsybakov: Model Margin Noise and $\mathcal{H}$-Consistency Bounds

TL;DR

The paper introduces Model Margin (MM) noise, a hypothesis-dependent low-noise condition that is weaker than the classical Tsybakov noise and can hold even when Tsybakov fails. It proves enhanced ${\mathscr H}$-consistency bounds for both binary and multi-class classification under MM, preserving the favorable exponents and yielding predictor-dependent constants via ${\mathbb E}[1_{\mu(h,X)>0}]^{1/t}$. A key lemma bounds the disagreement mass by a power of the 0-1 excess error, enabling the non-asymptotic transfer from surrogate to target excess risk under MM. The authors instantiate these bounds for common surrogate families (e.g., margin-based, cross-entropy, and comp-sum losses), and demonstrate structural properties such as monotonicity and invariance, broadening the applicability of enhanced ${\mathscr H}$-consistency results in practice.

Abstract

We introduce a new low-noise condition for classification, the Model Margin Noise (MM noise) assumption, and derive enhanced $\mathcal{H}$-consistency bounds under this condition. MM noise is weaker than Tsybakov noise condition: it is implied by Tsybakov noise condition but can hold even when Tsybakov fails, because it depends on the discrepancy between a given hypothesis and the Bayes-classifier rather than on the intrinsic distributional minimal margin (see Figure 1 for an illustration of an explicit example). This hypothesis-dependent assumption yields enhanced $\mathcal{H}$-consistency bounds for both binary and multi-class classification. Our results extend the enhanced $\mathcal{H}$-consistency bounds of Mao, Mohri, and Zhong (2025a) with the same favorable exponents but under a weaker assumption than the Tsybakov noise condition; they interpolate smoothly between linear and square-root regimes for intermediate noise levels. We also instantiate these bounds for common surrogate loss families and provide illustrative tables.

Figures (1)

• Figure 1: MM holds while Tsybakov fails. Consider $X \sim \mathrm{Unif}[0, 1]$ and $\eta(x) = \tfrac{1}{2} + c\, x^{\beta}$ (with $c \in (*){0, \tfrac{1}{2}}, \beta > 0$). The Bayes classifier is $h^*(x) \equiv +1$ and the minimal margin is $\gamma(x) = 2c\, x^{\beta}$. The Tsybakov tail is $\mathop{\mathrm{\mathbb{P}}}\limits [*]{\gamma(X) \leq t} = (*){ \tfrac{t}{2c} }^{1/\beta}$. If we choose $\beta > \tfrac{1 - \alpha}{\alpha}$, this tail is "heavy" and the Tsybakov condition fails for $\alpha$. However, if we use the restricted class ${\mathscr H} = \{*\}{h^*}$, the model margin is $\mu(h^*, x)\equiv 0$. Thus, $\mathop{\mathrm{\mathbb{P}}}\limits [*]{0 < \mu(h^*, X) \leq t} = 0$, and the MM noise condition holds trivially.

Theorems & Definitions (13)

• Lemma 0
• Definition 1: Model Margin (MM) noise
• Definition 2: Tsybakov noise mao2025enhanced
• theorem 3: MM $\not\Rightarrow$ Tsybakov
• proof
• Lemma 3: Disagreement mass vs. $0$–$1$ excess error
• proof
• theorem 4
• proof
• theorem 5