Faster Algorithms for Agnostically Learning Disjunctions and their Implications
Ilias Diakonikolas, Daniel M. Kane, Lisheng Ren
TL;DR
This work advances distribution-free agnostic PAC learning of disjunctions by presenting algorithms with time complexity $2^{\tilde{O}(n^{1/3})}$, substantially beating the previous $2^{\tilde{O}(n^{1/2})}$ bound achieved by $L_1$-polynomial regression. It develops both a sample-based and a Statistical Query (SQ) version, the latter achieving the same asymptotic bound and thereby establishing a super-polynomial separation between CSQ and SQ in agnostic learning. The approach hinges on partitioning the input space by Hamming weight, exploiting low-degree polynomial approximations on the light-weight region, and recursively pruning coordinates on the heavy-weight region; the heavy-coordinate mechanism yields a controlled reduction in problem size while maintaining correctness. These results illuminate qualitative differences between distribution-specific and distribution-free agnostic learning, and open avenues for faster agnostic algorithms for DNFs and related function classes.
Abstract
We study the algorithmic task of learning Boolean disjunctions in the distribution-free agnostic PAC model. The best known agnostic learner for the class of disjunctions over $\{0, 1\}^n$ is the $L_1$-polynomial regression algorithm, achieving complexity $2^{\tilde{O}(n^{1/2})}$. This complexity bound is known to be nearly best possible within the class of Correlational Statistical Query (CSQ) algorithms. In this work, we develop an agnostic learner for this concept class with complexity $2^{\tilde{O}(n^{1/3})}$. Our algorithm can be implemented in the Statistical Query (SQ) model, providing the first separation between the SQ and CSQ models in distribution-free agnostic learning.