Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibrated Principal Component Regression

Yixuan Florence Wu, Yilun Zhu, Lei Cao, Naichen Shi

TL;DR

Calibrated Principal Component Regression (CPCR) tackles truncation bias in PCR for generalized linear models in the overparameterized regime by learning a low-variance PC-based prior and calibrating in the full feature space through a centered Tikhonov step. The method employs cross-fitting to separate learning and calibration, and combines two calibrated estimators by averaging, thereby preserving variance reduction while mitigating bias. A random matrix theory analysis yields exact asymptotic risk limits, showing CPCR outperforms PCR when the predictive signal leaks into low-variance directions (κ < 1) and highlighting dependence on the nuisance spectrum rather than the exact informative subspace. Empirically, CPCR delivers consistent gains over PCR and ridge across synthetic data, Nyström-kernel regression, and vision-based classification tasks, demonstrating stability and robustness in modern high-dimensional learning tasks. The work provides a principled, scalable approach to debiasing latent-factor regression with strong theoretical guarantees and broad empirical validation.

Abstract

We propose a new method for statistical inference in generalized linear models. In the overparameterized regime, Principal Component Regression (PCR) reduces variance by projecting high-dimensional data to a low-dimensional principal subspace before fitting. However, PCR incurs truncation bias whenever the true regression vector has mass outside the retained principal components (PC). To mitigate the bias, we propose Calibrated Principal Component Regression (CPCR), which first learns a low-variance prior in the PC subspace and then calibrates the model in the original feature space via a centered Tikhonov step. CPCR leverages cross-fitting and controls the truncation bias by softening PCR's hard cutoff. Theoretically, we calculate the out-of-sample risk in the random matrix regime, which shows that CPCR outperforms standard PCR when the regression signal has non-negligible components in low-variance directions. Empirically, CPCR consistently improves prediction across multiple overparameterized problems. The results highlight CPCR's stability and flexibility in modern overparameterized settings.

Calibrated Principal Component Regression

TL;DR

Calibrated Principal Component Regression (CPCR) tackles truncation bias in PCR for generalized linear models in the overparameterized regime by learning a low-variance PC-based prior and calibrating in the full feature space through a centered Tikhonov step. The method employs cross-fitting to separate learning and calibration, and combines two calibrated estimators by averaging, thereby preserving variance reduction while mitigating bias. A random matrix theory analysis yields exact asymptotic risk limits, showing CPCR outperforms PCR when the predictive signal leaks into low-variance directions (κ < 1) and highlighting dependence on the nuisance spectrum rather than the exact informative subspace. Empirically, CPCR delivers consistent gains over PCR and ridge across synthetic data, Nyström-kernel regression, and vision-based classification tasks, demonstrating stability and robustness in modern high-dimensional learning tasks. The work provides a principled, scalable approach to debiasing latent-factor regression with strong theoretical guarantees and broad empirical validation.

Abstract

We propose a new method for statistical inference in generalized linear models. In the overparameterized regime, Principal Component Regression (PCR) reduces variance by projecting high-dimensional data to a low-dimensional principal subspace before fitting. However, PCR incurs truncation bias whenever the true regression vector has mass outside the retained principal components (PC). To mitigate the bias, we propose Calibrated Principal Component Regression (CPCR), which first learns a low-variance prior in the PC subspace and then calibrates the model in the original feature space via a centered Tikhonov step. CPCR leverages cross-fitting and controls the truncation bias by softening PCR's hard cutoff. Theoretically, we calculate the out-of-sample risk in the random matrix regime, which shows that CPCR outperforms standard PCR when the regression signal has non-negligible components in low-variance directions. Empirically, CPCR consistently improves prediction across multiple overparameterized problems. The results highlight CPCR's stability and flexibility in modern overparameterized settings.
Paper Structure (31 sections, 2 theorems, 18 equations, 7 figures, 2 tables)

This paper contains 31 sections, 2 theorems, 18 equations, 7 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Works
  3. Principal component regression (PCR).
  4. Ridge regression in high dimensions.
  5. Debiased regression and machine learning.
  6. PROBLEM FORMULATION
  7. METHOD
  8. Step 1: PCR.
  9. Step 2: calibration.
  10. Step 3: exchange and re-calibration.
  11. ANALYSIS OF CALIBRATION ESTIMATOR RISK
  12. Risk Analysis of CPCR
  13. Data generation process.
  14. Distributional assumptions on $\bm{\gamma}\xspace^*$.
  15. Risk.
  16. ...and 16 more sections

Key Result

Theorem 1

In the asymptotic limit where $n,p \rightarrow\infty$ such that $p/n \rightarrow c\in(1,\infty)$. With high probability, the risk of our calibration estimator $\hat{\bm{\gamma}}^{\mathtt{calib}}$ converge to a deterministic limit: where the bias term can be further estimated by and the variance term can be estimated by Moreover, consider $\{\mu_j(\bm{\Sigma}_c)\}_{j=1}^{p-r}$ the diagonal entri

Figures (7)

  • Figure 1: Data points with different membership, group 0 (grey dots) and group 1 (orange dots). The black arrow shows the top principal component. When considering the top PC only, PCR's decision boundary is influenced by truncation bias, while CPCR is able to calibrate its decision boundary.
  • Figure 2: Theoretical limits of the empirical risks. Error bars denote the empirical risks (\ref{['eq:risk']}) and solid curves denote the theoretical limits (\ref{['eq:limit_theory']}).
  • Figure 3: Risks of CPCR, PCR, and ridge regression. Eigenvalues of $\bm{\Sigma}_s$ are from $[2,4]$ and those of $\bm{\Sigma}_c$ are from $[1,3]$. CPCR consistently outperforms PCR and ridge regression.
  • Figure 4: Risks of CPCR, PCR, and ridge regression with different range of eigenvalues in $\bm{\Sigma}_c$ with $\kappa=0.995$. CPCR is more resistant to the increasing magnitudes of signals in the background space.
  • Figure 5: Normalized risk landscape over $(\kappa, \lambda)$. The white curve traces the optimal regularization parameter $\lambda^*$ for each $\kappa$. The background color represents the risk evaluated at each $(\kappa, \lambda)$ pair, devided by the optimal risk corresponding to that $\kappa$. Darker (purple) regions indicate risks closer to the optimum, while brighter (yellow) regions correspond to higher risks.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Definition 1
  • Theorem 1: Exact calibration estimator risk.
  • Theorem 2: Optimal $\lambda$.