Fundamental Novel Consistency Theory: $H$-Consistency Bounds
Yutao Zhong
TL;DR
This work develops a comprehensive, hypothesis-set–dependent theory of H-consistency bounds that quantify how closely a surrogate loss tracks a target loss across binary and multi-class classification tasks, including adversarial robustness. It introduces minimizability gaps to capture intrinsic limitations of a chosen hypothesis class, and provides universal transformation tools (Psi/Gamma and error-transformations) to derive tight, distribution-aware bounds for a broad family of surrogates, including comp-sum and constrained losses. The framework yields new, tight bounds for linear models and one-hidden-layer neural networks, extends to adversarial settings with both negative and positive results (notably favoring non-convex or margin-based surrogates like rho-margin), and validates the theory with simulations and large-scale experiments (e.g., CIFAR datasets). A key outcome is a universal square-root growth rate near zero for smooth surrogates in both binary and multi-class tasks, guiding surrogate selection through minimizability gaps and class-count considerations. The results provide principled guidance for designing and analyzing surrogate losses and learning algorithms with stronger, predictor- and distribution-aware guarantees, including robust learning paradigms.
Abstract
In machine learning, the loss functions optimized during training often differ from the target loss that defines task performance due to computational intractability or lack of differentiability. We present an in-depth study of the target loss estimation error relative to the surrogate loss estimation error. Our analysis leads to $H$-consistency bounds, which are guarantees accounting for the hypothesis set $H$. These bounds offer stronger guarantees than Bayes-consistency or $H$-calibration and are more informative than excess error bounds. We begin with binary classification, establishing tight distribution-dependent and -independent bounds. We provide explicit bounds for convex surrogates (including linear models and neural networks) and analyze the adversarial setting for surrogates like $ρ$-margin and sigmoid loss. Extending to multi-class classification, we present the first $H$-consistency bounds for max, sum, and constrained losses, covering both non-adversarial and adversarial scenarios. We demonstrate that in some cases, non-trivial $H$-consistency bounds are unattainable. We also investigate comp-sum losses (e.g., cross-entropy, MAE), deriving their first $H$-consistency bounds and introducing smooth adversarial variants that yield robust learning algorithms. We develop a comprehensive framework for deriving these bounds across various surrogates, introducing new characterizations for constrained and comp-sum losses. Finally, we examine the growth rates of $H$-consistency bounds, establishing a universal square-root growth rate for smooth surrogates in binary and multi-class tasks, and analyze minimizability gaps to guide surrogate selection.