Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization

Simone Bombari, Marco Mondelli

TL;DR

This work provides a precise statistical account of spurious correlations in high-dimensional regression, modeling inputs as a core feature $x$ and a spurious feature $y$, and quantifying the learned spurious signal via $\mathcal{C}$. For linear regression with ridge regularization, the authors derive a non-asymptotic, deterministic target $\mathcal{C}^{\Sigma}(\lambda)$ that $\mathcal{C}(\hat{\theta}_{LR}(\lambda))$ concentrates to, with the target expressed through the data covariance $\Sigma$ and its Schur complement relative to $x$. They show a trade-off between in-distribution test loss $\mathcal{L}$ and spurious correlations: the regularization strength $\lambda$ that minimizes $\mathcal{L}$ lies in an interval where $\mathcal{C}^{\Sigma}(\lambda)$ is increasing, implying that some spurious signal can be beneficial in-distribution. Extending to over-parameterized models, they prove an exact random-features equivalence to regularized LR with an effective $\tilde{\lambda}$ determined by the activation’s Hermite coefficients, which preserves spurious correlations in the RF setting. Together with experiments on Gaussian data, Color-MNIST, and CIFAR-10, the results illuminate how regularization, simplicity bias, and over-parameterization shape the reliance on spurious features and their impact on robustness and generalization.

Abstract

Learning models have been shown to rely on spurious correlations between non-predictive features and the associated labels in the training data, with negative implications on robustness, bias and fairness. In this work, we provide a statistical characterization of this phenomenon for high-dimensional regression, when the data contains a predictive core feature $x$ and a spurious feature $y$. Specifically, we quantify the amount of spurious correlations $C$ learned via linear regression, in terms of the data covariance and the strength $λ$ of the ridge regularization. As a consequence, we first capture the simplicity of $y$ through the spectrum of its covariance, and its correlation with $x$ through the Schur complement of the full data covariance. Next, we prove a trade-off between $C$ and the in-distribution test loss $L$, by showing that the value of $λ$ that minimizes $L$ lies in an interval where $C$ is increasing. Finally, we investigate the effects of over-parameterization via the random features model, by showing its equivalence to regularized linear regression. Our theoretical results are supported by numerical experiments on Gaussian, Color-MNIST, and CIFAR-10 datasets.

Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization

TL;DR

This work provides a precise statistical account of spurious correlations in high-dimensional regression, modeling inputs as a core feature and a spurious feature , and quantifying the learned spurious signal via . For linear regression with ridge regularization, the authors derive a non-asymptotic, deterministic target that concentrates to, with the target expressed through the data covariance and its Schur complement relative to . They show a trade-off between in-distribution test loss and spurious correlations: the regularization strength that minimizes lies in an interval where is increasing, implying that some spurious signal can be beneficial in-distribution. Extending to over-parameterized models, they prove an exact random-features equivalence to regularized LR with an effective determined by the activation’s Hermite coefficients, which preserves spurious correlations in the RF setting. Together with experiments on Gaussian data, Color-MNIST, and CIFAR-10, the results illuminate how regularization, simplicity bias, and over-parameterization shape the reliance on spurious features and their impact on robustness and generalization.

Abstract

Learning models have been shown to rely on spurious correlations between non-predictive features and the associated labels in the training data, with negative implications on robustness, bias and fairness. In this work, we provide a statistical characterization of this phenomenon for high-dimensional regression, when the data contains a predictive core feature and a spurious feature . Specifically, we quantify the amount of spurious correlations learned via linear regression, in terms of the data covariance and the strength of the ridge regularization. As a consequence, we first capture the simplicity of through the spectrum of its covariance, and its correlation with through the Schur complement of the full data covariance. Next, we prove a trade-off between and the in-distribution test loss , by showing that the value of that minimizes lies in an interval where is increasing. Finally, we investigate the effects of over-parameterization via the random features model, by showing its equivalence to regularized linear regression. Our theoretical results are supported by numerical experiments on Gaussian, Color-MNIST, and CIFAR-10 datasets.

Paper Structure

This paper contains 36 sections, 18 theorems, 163 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Spurious correlations.
  4. Simplicity bias.
  5. High-dimensional regression.
  6. Preliminaries
  7. Notation.
  8. Setting.
  9. Spurious correlations.
  10. Precise analysis for linear regression
  11. Warm-up: no regularization ($\lambda=0$).
  12. General case with regularization ($\lambda > 0$).
  13. Roles of regularization and simplicity bias
  14. Trade-off between $\mathcal{L} (\hat{\theta}_{\textup{LR}}(\lambda))$ and $\mathcal{C} (\hat{\theta}_{\textup{LR}}(\lambda))$.
  15. Role of Over-parameterization
  16. ...and 21 more sections

Key Result

Proposition 4.1

Let $\lambda = 0$ and $Z^\top Z \in \mathbb{R}^{2d \times 2d}$ be invertibleUnder Assumption ass:data, this holds with probability 1 for $n \geq 2d$.. Let $\mathcal{C}(\hat{\theta}_{\textup{LR}}(0))$ be the amount of spurious correlations learned by the model $f_{\textup{LR}}(\hat{\theta}_{\textup{L Furthermore, if Assumption ass:data holds and $n = \omega(d)$, with probability at least $1 - 2 \e

Figures (8)

  • Figure 1: Left two panels: pictorial representation of the core (spurious) feature $x$ ($y$) and an independent core feature $\tilde{x}$, taken from an image of a boat and a truck in the CIFAR-10 dataset. Right two panels: examples from a binary Color-MNIST dataset, where the labels correspond to the number shapes, and the zeros (ones) are colored in blue (red) with probability $(1 + \alpha)/2$.
  • Figure 2: Test loss $\mathcal{L}(\hat{\theta}_{\textup{LR}}(\lambda))$ (black) and spurious correlations $\mathcal{C}(\hat{\theta}_{\textup{LR}}(\lambda)$ (red) as a function of the regularization term $\lambda$ for two values of the number of samples $n$. Left: synthetic Gaussian dataset, with $d = 400$ (additional details in Appendix \ref{['app:experiments']}); right: binary Color-MNIST dataset with correlation $\sqrt{1 - \alpha^2} = 0.25$ between color and digit (see Figure \ref{['fig:intro']}).
  • Figure 3: Test loss $\mathcal{L}(\hat{\theta}_{\textup{LR}/\textup{RF}}(\lambda))$ (black) and spurious correlations $\mathcal{C}(\hat{\theta}_{\textup{LR}/\textup{RF}}(\lambda))$ (red) as a function of $\lambda_{\rm max}\left(\Sigma_{yy}\right)$ (left) and $\lambda_{\rm min}\left(S_x^\Sigma\right)$ (right) on a synthetic Gaussian dataset, for both linear regression and random features, with $\lambda = 1$ (additional details in Appendix \ref{['app:experiments']}).
  • Figure 4: Test loss $\mathcal{L}(\hat{\theta}_{\textup{LR}/\textup{RF}}(\lambda))$ (black) and spurious correlations $\mathcal{C}(\hat{\theta}_{\textup{LR}/\textup{RF}}(\lambda)$ (red) as a function of $\lambda_{\rm max}\left(\Sigma_{yy}\right) / \mathop{\rm tr}\nolimits(\Sigma_{yy})$ on a CIFAR-10 dataset for different levels of whitening (details on the whitening process in Appendix \ref{['app:experiments']}). We restrict to the classes "boat" and "truck" ($n = 10000$) and consider both linear regression and random features, with $\lambda = 1$.
  • Figure 5: Test loss $\mathcal{L}(\hat{\theta}_{\textup{NN}/\textup{RF}}(\lambda))$ (black) and spurious correlations $\mathcal{C}(\hat{\theta}_{\textup{NN}/\textup{RF}}(\lambda)$ (red) as a function of $\lambda$. Left: 2-layer fully connected ReLU network, trained on the binary color(C)-MNIST and CIFAR-10 (boats and trucks). Right: RF model with $\tanh$ and $\phi_1 = h_1 + 0.1 \, h_3$ activation. Implementation details are in Appendix \ref{['app:experiments']}.
  • ...and 3 more figures

Theorems & Definitions (28)

  • Proposition 4.1
  • Theorem 1
  • Proposition 5.1
  • Proposition 5.2
  • Proposition 5.3
  • Theorem 2
  • Proposition B.1
  • proof
  • Lemma B.2
  • proof
  • ...and 18 more