Learning Noisy Halfspaces with a Margin: Massart is No Harder than Random
Gautam Chandrasekaran, Vasilis Kontonis, Konstantinos Stavropoulos, Kevin Tian
TL;DR
This work addresses PAC learning of γ-margin halfspaces under Massart noise η, introducing Perspectron, a simple proper learner that matches the best sample complexity under random classification and extends to Massart GLMs with a known link σ. The core technique combines a certificate-based semi-random noise framework with an inverse-margin reweighting to produce a bounded separating hyperplane, enabling SGD-like updates and cutting-plane refinements. The authors show that ε-accurate learning (_error ≤ η+ε) is achievable with ~O((εγ)^{-2}) samples, and they extend the approach to σ-Massart GLMs with comparable guarantees, improving upon prior results in both models. They discuss limitations, open questions, and note concurrent independent work delivering essentially the same results, highlighting the robustness and potential practical impact of these semi-random-noise learning strategies.
Abstract
We study the problem of PAC learning $γ$-margin halfspaces with Massart noise. We propose a simple proper learning algorithm, the Perspectron, that has sample complexity $\widetilde{O}((εγ)^{-2})$ and achieves classification error at most $η+ε$ where $η$ is the Massart noise rate. Prior works [DGT19,CKMY20] came with worse sample complexity guarantees (in both $ε$ and $γ$) or could only handle random classification noise [DDK+23,KIT+23] -- a much milder noise assumption. We also show that our results extend to the more challenging setting of learning generalized linear models with a known link function under Massart noise, achieving a similar sample complexity to the halfspace case. This significantly improves upon the prior state-of-the-art in this setting due to [CKMY20], who introduced this model.