Learning Intersections of Halfspaces with Distribution Shift: Improved Algorithms and SQ Lower Bounds
Adam R. Klivans, Konstantinos Stavropoulos, Arsen Vasilyan
TL;DR
The paper tackles learning under distribution shift (TDS) for intersections of halfspaces with Gaussian training data, introducing efficient algorithms and SQ lower bounds that closely align with PAC-like performance under a mild ε-balanced assumption. The core strategy combines dimension reduction via subspace retrieval with carefully constructed candidate-hypothesis covers and localized testers to certify low test error from limited unlabeled test data, quantified through a discrepancy-distance framework. The authors deliver tight upper bounds for ε-balanced homogeneous intersections of k halfspaces, achieving runtime 2^{(k/ε)^{O(1)}} poly(d), and provide efficient but more intricate bounds for general and non-degenerate cases, together with first SQ lower bounds for any TDS problem. They further show that the ε-balanced assumption is necessary for poly(d,1/ε)-time TDS learning for a single halfspace and that even balanced intersections of two general halfspaces face d^{ ilde{Ω}(\log(1/ε))} SQ lower bounds, highlighting fundamental computational limits. Overall, the work broadens the toolkit for TDS learning—through subspace-recovery, covering arguments, and discrepancy-based certification—and sharpens our understanding of when distribution shift can be efficiently handled in high-dimensional, halfspace-based models.
Abstract
Recent work of Klivans, Stavropoulos, and Vasilyan initiated the study of testable learning with distribution shift (TDS learning), where a learner is given labeled samples from training distribution $\mathcal{D}$, unlabeled samples from test distribution $\mathcal{D}'$, and the goal is to output a classifier with low error on $\mathcal{D}'$ whenever the training samples pass a corresponding test. Their model deviates from all prior work in that no assumptions are made on $\mathcal{D}'$. Instead, the test must accept (with high probability) when the marginals of the training and test distributions are equal. Here we focus on the fundamental case of intersections of halfspaces with respect to Gaussian training distributions and prove a variety of new upper bounds including a $2^{(k/ε)^{O(1)}} \mathsf{poly}(d)$-time algorithm for TDS learning intersections of $k$ homogeneous halfspaces to accuracy $ε$ (prior work achieved $d^{(k/ε)^{O(1)}}$). We work under the mild assumption that the Gaussian training distribution contains at least an $ε$ fraction of both positive and negative examples ($ε$-balanced). We also prove the first set of SQ lower-bounds for any TDS learning problem and show (1) the $ε$-balanced assumption is necessary for $\mathsf{poly}(d,1/ε)$-time TDS learning for a single halfspace and (2) a $d^{\tildeΩ(\log 1/ε)}$ lower bound for the intersection of two general halfspaces, even with the $ε$-balanced assumption. Our techniques significantly expand the toolkit for TDS learning. We use dimension reduction and coverings to give efficient algorithms for computing a localized version of discrepancy distance, a key metric from the domain adaptation literature.