Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoisers

TL;DR

This work analyzes supervised denoising and noisy-input regression under distribution shift in a low-rank data setting within a proportional regime where $d/N=c+o(1)$. It derives data-dependent, instance-specific test-error expressions that depend on the training/test data spectra and their left singular-vector alignment, and shows that double descent persists under distribution shift due to implicit regularization from input noise. The authors demonstrate both benign and tempered overfitting regimes and reveal how data augmentation can modulate test error in-distribution and out-of-distribution. They validate the theory with real-life-like datasets, achieving sub-1% MSE accuracy in low-rank scenarios, and provide insights into OOD generalization, transfer learning, and the interplay between noise magnitude and distribution shift. Overall, the paper advances understanding of generalization for noisy inputs beyond IID, full-rank assumptions and offers practical guidance for augmentation and robustness in denoising tasks.

Abstract

Despite the importance of denoising in modern machine learning and ample empirical work on supervised denoising, its theoretical understanding is still relatively scarce. One concern about studying supervised denoising is that one might not always have noiseless training data from the test distribution. It is more reasonable to have access to noiseless training data from a different dataset than the test dataset. Motivated by this, we study supervised denoising and noisy-input regression under distribution shift. We add three considerations to increase the applicability of our theoretical insights to real-life data and modern machine learning. First, while most past theoretical work assumes that the data covariance matrix is full-rank and well-conditioned, empirical studies have shown that real-life data is approximately low-rank. Thus, we assume that our data matrices are low-rank. Second, we drop independence assumptions on our data. Third, the rise in computational power and dimensionality of data have made it important to study non-classical regimes of learning. Thus, we work in the non-classical proportional regime, where data dimension $d$ and number of samples $N$ grow as $d/N = c + o(1)$. For this setting, we derive data-dependent, instance specific expressions for the test error for both denoising and noisy-input regression, and study when overfitting the noise is benign, tempered or catastrophic. We show that the test error exhibits double descent under general distribution shift, providing insights for data augmentation and the role of noise as an implicit regularizer. We also perform experiments using real-life data, where we match the theoretical predictions with under 1\% MSE error for low-rank data.

Figures (11)

• Figure 1: Figures showing the test error for Linear Regression vs $1/c = N/d$. Training data from the CIFAR dataset is projected onto its first $r$ principal components for $r = 25,50,100,150$. 2500 test data points from CIFAR, STL10, and SVHN datasets are projected onto the same low-dimensional subspace. For empirical data points, shown by markers, we report the mean test error over at least 200 trials.
• Figure 2: Figures showing the test error for $\beta = I$ vs $1/c = N/d$. In (a) and (b), training data from the CIFAR dataset is projected onto its first $r$ principal components for $r = 25,50,100,150$. 2500 test data points from CIFAR (Green, Left col.), STL10 (Blue, Middle col.), and SVHN (Red, Right col.) datasets are projected onto the same low-dimensional subspace. (a) is in-subspace test error and (b) is out-of-subspace test error. In (c), we don't project the test data and report the standard test error, relying on the approximate low-rank structure in data instead of imposing it. For empirical data points, shown by markers, we report the mean test error over at least 200 trials. Similar results are obtained for single-variable regression with $\beta \in \mathbb{R}^d$.
• Figure 3: MSE and Accuracy when we only have covariate shift and $\beta_{tst} = \beta$.
• Figure 4: Experiment when we have both covariate shift and a shift in the target function.
• Figure 5: Optimal $\eta_{trn}$ that minimizes the test error given in Theorem \ref{['thm:main']} versus $c = d/N_{trn}$.

Theorems & Definitions (77)

• Definition 1
• Definition 2
• Definition 3
• Remark 1
• Remark 2
• Theorem 1: In-Subspace Test Error
• proof : Proof Sketch:
• Corollary 1: Distribution Shift Bound
• Theorem 2: Out-of-Subspace Shift Bound
• Theorem 3: Transfer Learning