Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On a synergistic learning phenomenon in nonparametric domain adaptation

Ling Zhou, Yuhong Yang

TL;DR

This work analyzes nonparametric regression under covariate shift with potentially unbounded likelihood ratios, deriving the minimax transfer learning rates when using both source and target data. A key contribution is the discovery of a synergistic learning phenomenon (SLP): under certain density-exponent regimes and when the target sample is not too small relative to the source, combining data yields rates faster than either dataset alone. The results cover known and unknown density parameters, extend to density singularities with transfer-singularity points, and include adaptive procedures for unknown smoothness $\beta$, backed by simulations that validate the theoretical rates. The findings illuminate how density inhomogeneity governs when and how source and target data can synergistically improve learning in domain adaptation for regression. The methods hinge on Hölder-smooth regression functions, Gaussian noise, and density-weighted risks $R_{h_T}$, with optimally tuned estimators (NW and local polynomials) and a spread function $t_n(x)$ governing local bandwidths.

Abstract

Consider nonparametric domain adaptation for regression, which assumes the same conditional distribution of the response given the covariates but different marginal distributions of the covariates. An important goal is to understand how the source data may improve the minimax convergence rate of learning the regression function when the likelihood ratio of the covariate marginal distributions of the target data and the source data are unbounded. A previous work of Pathak et al. (2022) show that the minimax transfer learning rate is simply determined by the faster rate of using either the source or the target data alone. In this paper, we present a new synergistic learning phenomenon (SLP) that the minimax convergence rate based on both data may sometimes be faster (even much faster) than the better rate of convergence based on the source or target data only. The SLP occurs when and only when the target sample size is smaller (in order) than but not too much smaller than the source sample size in relation to the smoothness of the regression function and the nature of the covariate densities of the source and target distributions. Interestingly, the SLP happens in two different ways according to the relationship between the two sample sizes. One is that the target data help alleviate the difficulty in estimating the regression function at points where the density of the source data is close to zero and the other is that the source data (with its larger sample size than that of the target data) help the estimation at points where the density of the source data is not small. Extensions to handle unknown source and target parameters and smoothness of the regression function are also obtained.

On a synergistic learning phenomenon in nonparametric domain adaptation

TL;DR

This work analyzes nonparametric regression under covariate shift with potentially unbounded likelihood ratios, deriving the minimax transfer learning rates when using both source and target data. A key contribution is the discovery of a synergistic learning phenomenon (SLP): under certain density-exponent regimes and when the target sample is not too small relative to the source, combining data yields rates faster than either dataset alone. The results cover known and unknown density parameters, extend to density singularities with transfer-singularity points, and include adaptive procedures for unknown smoothness , backed by simulations that validate the theoretical rates. The findings illuminate how density inhomogeneity governs when and how source and target data can synergistically improve learning in domain adaptation for regression. The methods hinge on Hölder-smooth regression functions, Gaussian noise, and density-weighted risks , with optimally tuned estimators (NW and local polynomials) and a spread function governing local bandwidths.

Abstract

Consider nonparametric domain adaptation for regression, which assumes the same conditional distribution of the response given the covariates but different marginal distributions of the covariates. An important goal is to understand how the source data may improve the minimax convergence rate of learning the regression function when the likelihood ratio of the covariate marginal distributions of the target data and the source data are unbounded. A previous work of Pathak et al. (2022) show that the minimax transfer learning rate is simply determined by the faster rate of using either the source or the target data alone. In this paper, we present a new synergistic learning phenomenon (SLP) that the minimax convergence rate based on both data may sometimes be faster (even much faster) than the better rate of convergence based on the source or target data only. The SLP occurs when and only when the target sample size is smaller (in order) than but not too much smaller than the source sample size in relation to the smoothness of the regression function and the nature of the covariate densities of the source and target distributions. Interestingly, the SLP happens in two different ways according to the relationship between the two sample sizes. One is that the target data help alleviate the difficulty in estimating the regression function at points where the density of the source data is close to zero and the other is that the source data (with its larger sample size than that of the target data) help the estimation at points where the density of the source data is not small. Extensions to handle unknown source and target parameters and smoothness of the regression function are also obtained.

Paper Structure

This paper contains 17 sections, 10 theorems, 63 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Minimax risks
  4. Transfer Learning Rate
  5. Minimax lower bounding
  6. Minimax upper bounding
  7. Optimal transfer learning rate and synergistic learning phenomenon
  8. Sample size relationship regions and SAR
  9. Simulation Studies
  10. Extension
  11. Density functions that may approach 0 at different orders
  12. An illustration
  13. Densities with unknown parameters
  14. Adaptation to the smoothness parameter $\beta$
  15. Concluding Remarks
  16. ...and 2 more sections

Key Result

Theorem 1

For any $a > 0$ and $\beta > 0$, the minimax learning risk satisfies: Here and hereafter, $c_j$ are positive constants, which are allowed to depend on $a$, $\beta$ and $\overline{\sigma}$, but are independent of $n$ and $n_T$. Furthermore, the above bounds hold for all choices of the constants $c_1, c_2, c_3$ and $c_4$. The inequality $A_{n, n_T} \succeq B_{n, n_T}$ m

Figures (3)

  • Figure 1: The logarithms of approximated SAR are plotted as $\log n$ increases, where the solid orange line represents the ratio calculated with $n_T = 0.1 \times n^{\frac{2\times 0.5 + 1}{2 \times 0.5 + 4}}$ (SSR-1), the dotted black line represents the ratio calculated with $n_T = 10\times n$ (SSR-4), the dashed red line represents the ratio calculated with $n_T = 0.5 \times n^{\frac{2\times 0.5 + 1}{2 \times 0.5 + 4}(1 + 0.95)}$ (SSR-2), and the dot-dashed blue line represents the ratio calculated with $n_T = n^{\frac{2\times 0.5 + 1}{2 \times 0.5 + 4}(1 + 1.2)}$ (SSR-3). In the first row, the true $f(x) = f_1(x) := 0.2x^{0.5}$. In the second row, the true $f(x) = f_2(x) := 0.2 |x - 0.5|^{0.5}$.
  • Figure 2: The logarithms of approximated SAR are plotted as $\log n$ increases, where the solid orange line represents the ratio calculated with $n_T = 0.1 \times n^{\frac{2\times 0.9 + 1}{2 \times 0.9 + 4}}$ (SSR-1), the dotted black line represents the ratio calculated with $n_T = 10\times n$ (SSR-4), the dashed red line represents the ratio calculated with $n_T = 0.5 \times n^{\frac{2\times 0.9 + 1}{2 \times 0.9 + 4}(1 + 0.4)}$ (SSR-2), and the dot-dashed blue line represents the ratio calculated with $n_T = n^{\frac{2\times 0.9 + 1}{2 \times 0.9 + 4}(1 + 0.7)}$ (SSR-3). In the first row, the true $f(x) = f_1(x) := 0.2x^{0.9}$. In the second row, the true $f(x) = f_2(x) := 0.2 |x - 0.5|^{0.9}$.
  • Figure 3: One example of the two density functions $(h_A, h_T)$ under two cases, where case (1): $h_A(x) = \frac{1}{0.005}\left( x^{3.5}(0.5 - x)I_{[0, 0.5]}(x) + (x - 0.5)^{4.5}I_{[0.5, 1]}(x)\right)$ (dotted red line), $h_T(x) = \frac{1}{0.172}\left(x^{0.5}(0.5-x)I_{[0, 0.5]}(x) + (x - 0.5)I_{[0.5, 1]}(x)\right)$ (dashed blue line); and case (2): $h_A(x) = \frac{1}{0.007}\left(x^{3.5}(0.5 - x)I_{[0, 0.5]}(x) + (x - 0.5)^4I_{[0.5, 1]}(x)\right)$ (dotted red line), $h_T(x) = \frac{1}{0.283}\left(x^{0.5}(0.5 - x)I_{[0, 0.5]}(x) + (x - 0.5)^{0.5}I_{0.5, 1}(x)\right)$ (dashed blue line).

Theorems & Definitions (10)

  • Theorem 1
  • Theorem 2
  • Corollary 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • Lemma A.7
  • Lemma A.8
  • Lemma A.9
  • Lemma A.10