Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards regularized learning from functional data with covariate shift

Markus Holzleitner, Sergiy Pereverzyev, Sergei V. Pereverzyev, Vaibhav Silmana, S. Sivananthan

TL;DR

The paper develops a regularized learning framework for vector-valued regression under covariate shift within the vector-valued reproducing kernel Hilbert space (vRKHS) setting. It introduces an operator-learning estimator that leverages importance weighting under covariate shift, together with a general regularization scheme and a KuLSIF-based estimation of the weight function $\beta$, and establishes optimal convergence rates under a general source condition $C_* = C_o \varphi(C_X)$ with eigen-decay $\mu_i \asymp i^{-b}$. To address tuning challenges, the authors propose an aggregation strategy over multiple regularization parameters and kernels, with a finite-sample analysis showing the aggregated predictor performs near the best constituent model. The approach is validated on a real-world face dataset with distributional shifts, demonstrating robustness to covariate shift and effective kernel selection via aggregation, and it extends minimax-optimal results for functional data to the covariate-shift setting.

Abstract

This paper investigates a general regularization framework for unsupervised domain adaptation in vector-valued regression under the covariate shift assumption, utilizing vector-valued reproducing kernel Hilbert spaces (vRKHS). Covariate shift occurs when the input distributions of the training and test data differ, introducing significant challenges for reliable learning. By restricting the hypothesis space, we develop a practical operator learning algorithm capable of handling functional outputs. We establish optimal convergence rates for the proposed framework under a general source condition, providing a theoretical foundation for regularized learning in this setting. We also propose an aggregation-based approach that forms a linear combination of estimators corresponding to different regularization parameters and different kernels. The proposed approach addresses the challenge of selecting appropriate tuning parameters, which is crucial for constructing a good estimator, and we provide a theoretical justification for its effectiveness. Furthermore, we illustrate the proposed method on a real-world face image dataset, demonstrating robustness and effectiveness in mitigating distributional discrepancies under covariate shift.

Towards regularized learning from functional data with covariate shift

TL;DR

The paper develops a regularized learning framework for vector-valued regression under covariate shift within the vector-valued reproducing kernel Hilbert space (vRKHS) setting. It introduces an operator-learning estimator that leverages importance weighting under covariate shift, together with a general regularization scheme and a KuLSIF-based estimation of the weight function , and establishes optimal convergence rates under a general source condition with eigen-decay . To address tuning challenges, the authors propose an aggregation strategy over multiple regularization parameters and kernels, with a finite-sample analysis showing the aggregated predictor performs near the best constituent model. The approach is validated on a real-world face dataset with distributional shifts, demonstrating robustness to covariate shift and effective kernel selection via aggregation, and it extends minimax-optimal results for functional data to the covariate-shift setting.

Abstract

This paper investigates a general regularization framework for unsupervised domain adaptation in vector-valued regression under the covariate shift assumption, utilizing vector-valued reproducing kernel Hilbert spaces (vRKHS). Covariate shift occurs when the input distributions of the training and test data differ, introducing significant challenges for reliable learning. By restricting the hypothesis space, we develop a practical operator learning algorithm capable of handling functional outputs. We establish optimal convergence rates for the proposed framework under a general source condition, providing a theoretical foundation for regularized learning in this setting. We also propose an aggregation-based approach that forms a linear combination of estimators corresponding to different regularization parameters and different kernels. The proposed approach addresses the challenge of selecting appropriate tuning parameters, which is crucial for constructing a good estimator, and we provide a theoretical justification for its effectiveness. Furthermore, we illustrate the proposed method on a real-world face image dataset, demonstrating robustness and effectiveness in mitigating distributional discrepancies under covariate shift.
Paper Structure (8 sections, 17 theorems, 150 equations, 8 figures, 2 tables)

This paper contains 8 sections, 17 theorems, 150 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Construction of Operator Learning Algorithm for Covariate shift
  4. LEARNING RATES AND ERROR BOUNDS
  5. Aggregation in Regularized Vector-Valued Learning with covariate shift
  6. Numerical Experiments
  7. Proofs of Auxiliary Results
  8. ACKNOWLEDGMENT

Key Result

Theorem 5

Let $\phi:X\rightarrow \mathcal{H}$ denotes the feature map such that $\phi(x) = k(x,\cdot)$. Then for every function $f \in \mathcal{G}$ there exists a unique operator $C \in \mathcal{S}_2(\mathcal{H}, {Y})$ such that with and vice versa. Hence $\mathcal{G} \simeq \mathcal{S}_2(\mathcal{H}, {Y})$ and it follows that $\mathcal{G}$ can be written as

Figures (8)

  • Figure 1: An illustration of Case 1: a face image converted to its sinogram, followed by application of a motion blur kernel.
  • Figure 2: An illustration of Case 1: a face image converted to its sinogram, followed by application of a Gaussian blur kernel.
  • Figure 3: An illustration of Case 2: motion blur applied directly to the face image prior to generating its sinogram.
  • Figure 4: An illustration of Case 2: Gaussian blur applied directly to the face image prior to generating its sinogram.
  • Figure 5: Reconstruction results under motion blur applied to the sinograms.
  • ...and 3 more figures

Theorems & Definitions (25)

  • Definition 1
  • Remark 2
  • Definition 3
  • Definition 4
  • Theorem 5: VRKHS_REF, Theorem 1
  • Proposition 6
  • Proposition 7: Representer theorem for spectral filter with importance weights
  • Lemma 8
  • Lemma 9
  • Lemma 10
  • ...and 15 more