Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Minimax Theory of Nonparametric Regression Under Covariate Shift

Petr Zamolodtchikov

Abstract

We consider nonparametric regression under covariate shift, where we observe samples from both the target distribution and a related but distinct source distribution. We introduce a novel object, the transfer function, and show that properties of its domain determine our minimax rates. Those exhibit a variety of regimes, including classical rates, governed by the better of source-only and target-only rates, as well as regimes in which the convergence rates exhibit multiplicative interactions between the sample sizes and are faster than the best-of-two benchmark. The rates are shown to be achieved up to logarithmic factors by a design-adaptive estimator. Compared with existing theory, our results cover the case in which covariates have unbounded support.

A Minimax Theory of Nonparametric Regression Under Covariate Shift

Abstract

We consider nonparametric regression under covariate shift, where we observe samples from both the target distribution and a related but distinct source distribution. We introduce a novel object, the transfer function, and show that properties of its domain determine our minimax rates. Those exhibit a variety of regimes, including classical rates, governed by the better of source-only and target-only rates, as well as regimes in which the convergence rates exhibit multiplicative interactions between the sample sizes and are faster than the best-of-two benchmark. The rates are shown to be achieved up to logarithmic factors by a design-adaptive estimator. Compared with existing theory, our results cover the case in which covariates have unbounded support.
Paper Structure (23 sections, 26 theorems, 267 equations, 4 figures)

This paper contains 23 sections, 26 theorems, 267 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Main Results
  4. Learnability and impact of covariate shift
  5. Analysis of the rates
  6. Phase diagrams
  7. Phase transition of transferability indices
  8. Samples sizes paths
  9. Heuristics, related works and examples
  10. Transfer function, estimator, and local mass
  11. Transfer function
  12. Local nearest neighbours regressor
  13. Local mass assumptions
  14. Preliminary Results
  15. Uniform control on neighbours
  16. ...and 8 more sections

Key Result

Theorem 4

Let $\operatorname{P}_{\mathrm{X}}, \operatorname{Q}\mathstrut_{\!\mathrm{X}} \in \mathcal{P}(D, \theta).$ Then, there exists an estimator $\widehat{f},$ an integer $N = N(\theta, D)$ and a constant $C(\tau) = C(\tau, d, \beta, \theta, \gamma^*, s^*)$, such that for all $\tau > 1,$ all $m\wedge n \g where the rate $\mathcal{R} = \mathcal{R}(n, m, \gamma, s)$ is defined as

Figures (4)

  • Figure 1: Left: Phase diagram in the $(\gamma, s)$ coordinates system. The dashed regions correspond to supercritical configurations. Center: Phase diagram in $(n, m)$ coordinates for $(\gamma, s) \in A(+, -).$Right: Phase diagram in $(n,m)$ coordinates for $(\gamma, s) \in A(-, +).$ The dashed regions correspond to the acceleration regime.
  • Figure 2: Subdivision of $(\gamma, s)$ phases for fixed $(n,m)$ pair. Here, $n < m.$
  • Figure 3: Phase transition in $(\log n, \log m)$ coordinates with $\gamma/r_\beta$ fixed. Left:$r_\beta < s < \gamma,$ subcritical configuration. Centre:$r_\beta = s < \gamma,$ critical configuration. Right:$s < r_\beta < \gamma,$ supercritical configuration.
  • Figure 4: Left: Two paths in $(\log n, \log m)$ coordinates. The linear path corresponds to $(n, m) = (B^{1-\lambda}, B^{\lambda})$ and the concave path corresponds to $(n, m) = ((1 - \lambda)B, \lambda B).$Centre: evolution of the logarithm of the rates along the linear path as a function of $\lambda.$Right: evolution of the logarithm of the rates along the fixed budget path as a function of $\lambda.$

Theorems & Definitions (59)

  • Definition 1: Transfer function
  • Definition 2: Integrability index
  • Definition 3
  • Theorem 4: Upper bound
  • Corollary 5: Upper bound, pure transfer
  • Theorem 6
  • Example 7: Pareto source-target pairs
  • Example 8: Exponential source-target pairs
  • Lemma 9
  • Lemma 10
  • ...and 49 more