Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Local convergence rates of the nonparametric least squares estimator with applications to transfer learning

Johannes Schmidt-Hieber, Petr Zamolodtchikov

TL;DR

A new indirect proof technique is developed that establishes the local convergence behavior based on a carefully chosen local perturbation of the LSE that implies that although least squares is a global criterion, the LSE adapts locally to the size of the design density.

Abstract

Convergence properties of empirical risk minimizers can be conveniently expressed in terms of the associated population risk. To derive bounds for the performance of the estimator under covariate shift, however, pointwise convergence rates are required. Under weak assumptions on the design distribution, it is shown that least squares estimators (LSE) over 1-Lipschitz functions are also minimax rate optimal with respect to a weighted uniform norm, where the weighting accounts in a natural way for the non-uniformity of the design distribution. This implies that although least squares is a global criterion, the LSE adapts locally to the size of the design density. We develop a new indirect proof technique that establishes the local convergence behavior based on a carefully chosen local perturbation of the LSE. The obtained local rates are then applied to analyze the LSE for transfer learning under covariate shift.

Local convergence rates of the nonparametric least squares estimator with applications to transfer learning

TL;DR

A new indirect proof technique is developed that establishes the local convergence behavior based on a carefully chosen local perturbation of the LSE that implies that although least squares is a global criterion, the LSE adapts locally to the size of the design density.

Abstract

Convergence properties of empirical risk minimizers can be conveniently expressed in terms of the associated population risk. To derive bounds for the performance of the estimator under covariate shift, however, pointwise convergence rates are required. Under weak assumptions on the design distribution, it is shown that least squares estimators (LSE) over 1-Lipschitz functions are also minimax rate optimal with respect to a weighted uniform norm, where the weighting accounts in a natural way for the non-uniformity of the design distribution. This implies that although least squares is a global criterion, the LSE adapts locally to the size of the design density. We develop a new indirect proof technique that establishes the local convergence behavior based on a carefully chosen local perturbation of the LSE. The obtained local rates are then applied to analyze the LSE for transfer learning under covariate shift.
Paper Structure (17 sections, 30 theorems, 225 equations, 4 figures)

This paper contains 17 sections, 30 theorems, 225 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Main results
  3. Local doubling property and examples of local rates
  4. Proof strategy
  5. Applications to Transfer Learning
  6. Using LSE from source distribution to predict under target distribution
  7. Combining both samples to predict under the target distribution
  8. Discussion
  9. A brief review of convergence results for the least squares estimator in nonparametric regression
  10. Related work on transfer learning
  11. Extensions and open problems
  12. Proof of the local convergence rate for the LSE
  13. Proofs for the properties of the spread function
  14. Proofs for upper bounds in Section 2
  15. Proofs for Section 3
  16. ...and 2 more sections

Key Result

Lemma 1

If $\operatorname{P}_\mathrm{X}\in \mathcal{M}$, then, for any $n >1$ and any $x \in [0, 1],$ there exists a unique solution $t_n(x)$ of the equation Therefore the function $x\mapsto t_n(x)$ is well defined on $[0,1]$. From now on, we refer to $t_n$ as the spread function (associated to $\operatorname{P}_\mathrm{X}$).

Figures (4)

  • Figure 1: The density $p_n$
  • Figure 2: If the LSE $\widehat{f}$ would not have locally slope $=1$, then one could construct a perturbed version $\tilde{f}$ that better fits the data, implying that $\widehat{f}$ cannot be a least squares fit.
  • Figure 3: (Construction of the local perturbation) The variables $x^*$ and $\tilde{x}$ are as in Lemma \ref{['lem.g_function']}. From the construction of $\tilde{x},$ we know that the function $\psi-f$ (plotted in blue) cannot lie above the green dashed curve with slope $\delta/2.$ The yellow function is $h_n - f.$ Since this function has slope $\delta$, it will hit the green dashed curve in a neighbourhood of $\tilde{x}$. This also implies that $h_n - f$ intersects for the first time with $\psi-f$ (blue curve) in this neighbourhood and provides us with control for the hitting points $x_\ell$ and $x_u.$ The (shifted) perturbation $g_n - f$ is given by the red curve.
  • Figure 4: If one point of $g$ is more than $r$ away from $h(x_i)$ on the interval $[x_i, x_{i+1}),$ then, by construction, $h$ moves in this direction on the interval $[x_{i+1},x_{i+2})$.

Theorems & Definitions (60)

  • Lemma 1
  • Definition 2
  • Lemma 3
  • Theorem 4
  • Corollary 5
  • Lemma 6
  • Theorem 7
  • Theorem 8
  • Lemma 9
  • Example 1
  • ...and 50 more