Sample-Near-Optimal Agnostic Boosting with Improved Running Time
Arthur da Cunha, Miakel Møller Høgsgaard, Andrea Paudice
TL;DR
This work closes the gap between near-optimal sample complexity and computational efficiency in agnostic boosting by presenting the first algorithm that achieves near-optimal sample complexity while running in time polynomial in the sample size for fixed problem parameters. It constructs a rich set of weak hypotheses via repeated subsampling and then selects a near-optimal combination by evaluating sign-averages on a held-out set, achieving an excess risk of $\mathrm{err}^{*}$ plus a sublinear term, specifically $\tilde{O}\big(\sqrt{\mathrm{err}^{*} \cdot d'/n} + d'/n\big)$ with $d' = O(T d \ln T)$ and $T = O(\min\{d^{*},\ln n\}/\theta^{2})$. The analysis blends uniform convergence over a $T$-wise averaged hypothesis class with Bernstein-type concentration, enabling a polynomial-time runtime despite the combinatorial potential of hypothesis combinations. The results approach the information-theoretic lower bounds (up to logarithmic factors) and offer practical guarantees for agnostic boosting under noisy or non-realizable data conditions.
Abstract
Boosting is a powerful method that turns weak learners, which perform only slightly better than random guessing, into strong learners with high accuracy. While boosting is well understood in the classic setting, it is less so in the agnostic case, where no assumptions are made about the data. Indeed, only recently was the sample complexity of agnostic boosting nearly settled arXiv:2503.09384, but the known algorithm achieving this bound has exponential running time. In this work, we propose the first agnostic boosting algorithm with near-optimal sample complexity, running in time polynomial in the sample size when considering the other parameters of the problem fixed.