Beyond Worst-Case Online Classification: VC-Based Regret Bounds for Relaxed Benchmarks
Omar Montasser, Abhishek Shetty, Nikita Zhivotovskiy
TL;DR
The paper reframes online binary classification by replacing worst-case optimality with relaxed benchmarks that capture robustness to input perturbations and smoothing, achieving regret bounds that scale with the VC dimension rather than the Littlestone dimension and feature only logarithmic dependence on the generalized margin. By constructing finite covers of the hypothesis space and applying Multiplicative Weights, the authors obtain regret guarantees against worst-case perturbations, Gaussian smoothing, and margin-based benchmarks, with matching lower bounds illustrating the necessity of the stated dependencies. A specialized treatment for halfspaces yields a tighter $O(\\sqrt{T d \, \\log(1/\\gamma)})$ bound, improving over generic dimension-dependent bounds. The work connects smoothed online learning and adversarial robustness ideas, showing how relaxed benchmarks can lead to computational and statistical benefits, and it relates to partial concept classes and private learning. Overall, the paper advances understanding of competitive performance under relaxed optimality notions, with implications for robust online prediction and efficient learning under perturbations.
Abstract
We revisit online binary classification by shifting the focus from competing with the best-in-class binary loss to competing against relaxed benchmarks that capture smoothed notions of optimality. Instead of measuring regret relative to the exact minimal binary error -- a standard approach that leads to worst-case bounds tied to the Littlestone dimension -- we consider comparing with predictors that are robust to small input perturbations, perform well under Gaussian smoothing, or maintain a prescribed output margin. Previous examples of this were primarily limited to the hinge loss. Our algorithms achieve regret guarantees that depend only on the VC dimension and the complexity of the instance space (e.g., metric entropy), and notably, they incur only an $O(\log(1/γ))$ dependence on the generalized margin $γ$. This stands in contrast to most existing regret bounds, which typically exhibit a polynomial dependence on $1/γ$. We complement this with matching lower bounds. Our analysis connects recent ideas from adversarial robustness and smoothed online learning.