Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computational Efficiency under Covariate Shift in Kernel Ridge Regression

Andrea Della Vecchia, Arnaud Mavakala Watusadisi, Ernesto De Vito, Lorenzo Rosasco

TL;DR

The paper tackles covariate shift in RKHS-based nonparametric regression by combining importance weighting with Nyström-based random subspaces to dramatically improve computational scalability. It develops a theoretically principled framework with high-probability excess risk bounds that hold even when IW weights may be unbounded, and shows that carefully chosen subspace sizes and regularization recover the best-known rates for kernel ridge regression under covariate shift. The main contributions include (i) deriving novel risk bounds for Nyström approximations under distribution mismatch, (ii) extending the analysis to unknown weights via a weight-vs-proxy decomposition, and (iii) validating the approach with simulations and real-data benchmarks where W-Nyström KRR achieves the accuracy of full weighted KRR with substantial time and memory savings. The results demonstrate that random projection techniques can yield scalable, statistically optimal kernel methods in covariate-shift settings, enabling practical deployment on large datasets.

Abstract

This paper addresses the covariate shift problem in the context of nonparametric regression within reproducing kernel Hilbert spaces (RKHSs). Covariate shift arises in supervised learning when the input distributions of the training and test data differ, presenting additional challenges for learning. Although kernel methods have optimal statistical properties, their high computational demands in terms of time and, particularly, memory, limit their scalability to large datasets. To address this limitation, the main focus of this paper is to explore the trade-off between computational efficiency and statistical accuracy under covariate shift. We investigate the use of random projections where the hypothesis space consists of a random subspace within a given RKHS. Our results show that, even in the presence of covariate shift, significant computational savings can be achieved without compromising learning performance.

Computational Efficiency under Covariate Shift in Kernel Ridge Regression

TL;DR

The paper tackles covariate shift in RKHS-based nonparametric regression by combining importance weighting with Nyström-based random subspaces to dramatically improve computational scalability. It develops a theoretically principled framework with high-probability excess risk bounds that hold even when IW weights may be unbounded, and shows that carefully chosen subspace sizes and regularization recover the best-known rates for kernel ridge regression under covariate shift. The main contributions include (i) deriving novel risk bounds for Nyström approximations under distribution mismatch, (ii) extending the analysis to unknown weights via a weight-vs-proxy decomposition, and (iii) validating the approach with simulations and real-data benchmarks where W-Nyström KRR achieves the accuracy of full weighted KRR with substantial time and memory savings. The results demonstrate that random projection techniques can yield scalable, statistically optimal kernel methods in covariate-shift settings, enabling practical deployment on large datasets.

Abstract

This paper addresses the covariate shift problem in the context of nonparametric regression within reproducing kernel Hilbert spaces (RKHSs). Covariate shift arises in supervised learning when the input distributions of the training and test data differ, presenting additional challenges for learning. Although kernel methods have optimal statistical properties, their high computational demands in terms of time and, particularly, memory, limit their scalability to large datasets. To address this limitation, the main focus of this paper is to explore the trade-off between computational efficiency and statistical accuracy under covariate shift. We investigate the use of random projections where the hypothesis space consists of a random subspace within a given RKHS. Our results show that, even in the presence of covariate shift, significant computational savings can be achieved without compromising learning performance.

Paper Structure

This paper contains 26 sections, 10 theorems, 91 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Classical Setting
  4. Covariate Shift Setting
  5. Importance-Weighting (IW) Correction
  6. Computations
  7. ERM on Random Subspaces and the Nyström Method
  8. Computations
  9. Statistical Guarantees
  10. Further Assumptions
  11. Excess Risk Bounds
  12. Unknown Weights
  13. Well Specified Case
  14. Misspecified Case
  15. Simulations and real data experiments
  16. ...and 11 more sections

Key Result

Theorem 1

Under assumptions H RKHS K bounded,ass:f_H exists,ass:bounded_moments,ass:bounded y,ass:source_cond_davit,ass:capacity cond, for ALS sampling, let $\delta>0$, $\left(\frac{256(W+\sigma^2)\log^2(4/\delta)}{n}\right)^{\frac{1}{\gamma(1-q)+1+q}}\leqslant \lambda\leqslant\|\Sigma\|_{op}$, and $m\geqslan

Figures (2)

  • Figure 1: On the left: MSE for the different models varying the number of train samples $n$. The result for W-Nyström KRR model is obtained for optimal $m$. On the right: $n=3000$, optimal $m$ is selected as the smallest for which W-Nyström KRR matches the full W-KRR model ($m=1000$ here).
  • Figure 2: The graphs above illustrate the results summarized in Table \ref{['tab:PPer']} for HHAR, HAR70+ and HARChildren datasets.

Theorems & Definitions (23)

  • Definition 1: Effective dimension
  • Theorem 1
  • Corollary 1
  • Example 1
  • Lemma 1: Nyström IW-KRR estimator
  • proof
  • Lemma 2: Nyström IW-KRR estimator, representer theorem form
  • proof
  • Definition 2: $T$-approximate leverage scores
  • Proposition 1: Empirical Effective Dimension
  • ...and 13 more