Tight Margin-Based Generalization Bounds for Voting Classifiers over Finite Hypothesis Sets

TL;DR

This work addresses margin-based generalization bounds for voting classifiers built from a finite base class $\\Hc$. By discretizing any voting classifier $f$ to a small, unweighted average $g$ within $\\Cc_N$ and leveraging the LS25 framework with Rademacher complexity, the authors derive an asymptotically tight bound that matches the known lower bounds across the margin $\\theta$, margin loss $\\Lc_\\Sr^\\theta(f)$, sample size $n$, hypothesis size $|\\Hc|$, and failure probability $\\delta$. A careful analysis of the discretization cost via Lipschitz-continuous surrogate functions $\\phi$ and $\\rho$, together with a partitioning of parameter ranges and a union bound over blocks, yields a bound of the form $\\Lc_\\Dc(f) \\le \\\Lc_\\Sr^\\theta(f) + c( \,\\sqrt{\\Lc_\\Sr^\\theta(f) [\\ln(e/\\Lc_\\Sr^\\theta(f)) \\\ln|\\Hc| / (\\theta^2 n) + \\ln(e/\\delta)/n]} + \\frac{\\ln(\\theta^2 n / \\ln|\\Hc|) \\ln|\\Hc|}{\\theta^2 n} + \\frac{\\ln(e/\\delta)}{n} )$, establishing asymptotic tightness for finite $\\Hc$. The result extends margin-based analyses to a broader, discretizable setting and clarifies the interplay between margin, training loss, and ensemble size for tight generalization guarantees. This tight bound strengthens theoretical understanding of AdaBoost-like voting classifiers and opens avenues for applying LS25-style techniques to other large-margin ensembles.

Abstract

We prove the first margin-based generalization bound for voting classifiers, that is asymptotically tight in the tradeoff between the size of the hypothesis set, the margin, the fraction of training points with the given margin, the number of training samples and the failure probability.

Theorems & Definitions (23)

• theorem 1
• lemma 1
• lemma 2
• lemma 3
• lemma 4
• lemma 5
• proof
• lemma 6
• lemma 7
• lemma 8