Benign Overfitting in Out-of-Distribution Generalization of Linear Models
Shange Tang, Jiayun Wu, Jianqing Fan, Chi Jin
TL;DR
The paper addresses how benign overfitting extends to Out-of-Distribution generalization for over-parameterized linear models under covariate shift. It derives non-asymptotic excess-risk bounds for ridge regression under general target covariances, identifying key quantities that govern performance and showing benign overfitting when the target’s minor-direction variance is controlled. The work introduces a sharp, instance-dependent framework that recovers known in-distribution bounds and under-parameterized OOD results as special cases, and demonstrates that Principal Component Regression can achieve the fast rate O(1/n) even when ridge does not, provided the true signal largely lies in the major directions. Simulations validate the theoretical rates, highlighting when ridge suffices and when PCR provides a robust alternative under large shifts in minor directions. Overall, the results illuminate the spectral conditions under which over-parameterized models generalize well under distribution shift and offer practical guidance for algorithm choice in OOD settings.
Abstract
Benign overfitting refers to the phenomenon where an over-parameterized model fits the training data perfectly, including noise in the data, but still generalizes well to the unseen test data. While prior work provides some theoretical understanding of this phenomenon under the in-distribution setup, modern machine learning often operates in a more challenging Out-of-Distribution (OOD) regime, where the target (test) distribution can be rather different from the source (training) distribution. In this work, we take an initial step towards understanding benign overfitting in the OOD regime by focusing on the basic setup of over-parameterized linear models under covariate shift. We provide non-asymptotic guarantees proving that benign overfitting occurs in standard ridge regression, even under the OOD regime when the target covariance satisfies certain structural conditions. We identify several vital quantities relating to source and target covariance, which govern the performance of OOD generalization. Our result is sharp, which provably recovers prior in-distribution benign overfitting guarantee [Tsigler and Bartlett, 2023], as well as under-parameterized OOD guarantee [Ge et al., 2024] when specializing to each setup. Moreover, we also present theoretical results for a more general family of target covariance matrix, where standard ridge regression only achieves a slow statistical rate of $O(1/\sqrt{n})$ for the excess risk, while Principal Component Regression (PCR) is guaranteed to achieve the fast rate $O(1/n)$, where $n$ is the number of samples.