Learning Neural Networks with Distribution Shift: Efficiently Certifiable Guarantees
Gautam Chandrasekaran, Adam R. Klivans, Lin Lin Lee, Konstantinos Stavropoulos
TL;DR
This work advances learning neural networks under distribution shift by formalizing a regression-appropriate Testable Learning with Distribution Shift (TDS) framework and delivering provably efficient algorithms that operate without assumptions on the test distribution. It fuses kernel methods with data-dependent feature maps and a spectral verification step to certify transfer from the training to the test distribution, enabling nonconvex regression with Lipschitz activations. For bounded, hypercontractive training marginals, it achieves fully polynomial-time learning for one-hidden-layer sigmoid nets and extends to deeper Lipschitz nets; for unbounded marginals with strictly subexponential tails, it uses uniform polynomial approximations and moment-based tests to obtain comparable guarantees. Overall, the paper provides practical, theory-backed algorithms for reliable learning under distribution shift, with explicit runtime and sample-complexity bounds tied to kernel representations and approximation theory.
Abstract
We give the first provably efficient algorithms for learning neural networks with distribution shift. We work in the Testable Learning with Distribution Shift framework (TDS learning) of Klivans et al. (2024), where the learner receives labeled examples from a training distribution and unlabeled examples from a test distribution and must either output a hypothesis with low test error or reject if distribution shift is detected. No assumptions are made on the test distribution. All prior work in TDS learning focuses on classification, while here we must handle the setting of nonconvex regression. Our results apply to real-valued networks with arbitrary Lipschitz activations and work whenever the training distribution has strictly sub-exponential tails. For training distributions that are bounded and hypercontractive, we give a fully polynomial-time algorithm for TDS learning one hidden-layer networks with sigmoid activations. We achieve this by importing classical kernel methods into the TDS framework using data-dependent feature maps and a type of kernel matrix that couples samples from both train and test distributions.