Error Guarantees for Least Squares Approximation with Noisy Samples in Domain Adaptation

TL;DR

This work analyzes weighted least-squares reconstruction of complex-valued functions from noisy samples drawn from a source measure that may differ from the target error measure, i.e., a domain-adaptation setting. It derives high-probability $L_2$ and $L_\infty$ error bounds that decompose into truncation, discretization, and noise terms, with a bias-variance trade-off governed by the approximation space dimension $m$ and oversampling. By employing concentration inequalities and the Christoffel function, the authors show logarithmic oversampling suffices for $L_2$-optimality in noiseless cases, while noise introduces a linear-in-$m$ term that must be balanced. They propose stable $H^1$ and $H^2$ Sobolev bases to achieve quadratic convergence and demonstrate the theory on unit interval and unit cube, including Sobolev spaces with dominating mixed smoothness, with numerical evidence supporting applicability in higher dimensions.

Abstract

Given $n$ samples of a function $f\colon D\to\mathbb C$ in random points drawn with respect to a measure $\varrho_S$ we develop theoretical analysis of the $L_2(D, \varrho_T)$-approximation error. For a parituclar choice of $\varrho_S$ depending on $\varrho_T$, it is known that the weighted least squares method from finite dimensional function spaces $V_m$, $\dim(V_m) = m < \infty$ has the same error as the best approximation in $V_m$ up to a multiplicative constant when given exact samples with logarithmic oversampling. If the source measure $\varrho_S$ and the target measure $\varrho_T$ differ we are in the domain adaptation setting, a subfield of transfer learning. We model the resulting deterioration of the error in our bounds. Further, for noisy samples, our bounds describe the bias-variance trade off depending on the dimension $m$ of the approximation space $V_m$. All results hold with high probability. For demonstration, we consider functions defined on the $d$-dimensional cube given in unifom random samples. We analyze polynomials, the half-period cosine, and a bounded orthonormal basis of the non-periodic Sobolev space $H_{\mathrm{mix}}^2$. Overcoming numerical issues of this $H_{\text{mix}}^2$ basis, this gives a novel stable approximation method with quadratic error decay. Numerical experiments indicate the applicability of our results.

Figures (5)

• Figure 1.1: One-dimensional approximation on the unit-interval for three different choices of $m$ and the schematic behaviour of the $L_2$-approximation error $\|f-S_m\bm y\|_{L_2}^2$ (solid line) split into the error for exact function values $\|f-S_m\bm f\|_{L_2}^2$ and the noise error $\|S_m\bm\varepsilon\|_{L_2}^2$ (dashed lines) with respect to $m$.
• Figure 4.1: One-dimensional experiment for different choices of $V_m$. Top row: minimal and maximal singular value of $1/\sqrt n \bm W^{1/2}\bm L$. Bottom row: the $L_2$-approximation error $\|f-S_m\bm y\|_{L_2}^2$ (solid line) split into the error for exact function values $\|f-S_m\bm f\|_{L_2}^2$ and the noise error $\|S_m\bm\varepsilon\|_{L_2}^2$ (dashed lines) with respect to $m$.
• Figure 4.2: Hyperbolic cross in three dimensions.
• Figure 4.3: Five-dimensional experiment for $H_{\mathrm{mix}}^2$. The solid lines represent the $L_2$-error $\|f-S_m\bm y\|_{ L_2}^2$ and the dashed lines the bound from Theorems \ref{['L2wo']} and \ref{['L2w']}.
• Figure A.1: $\cos(t)$ and $1/\cosh(t)$

Theorems & Definitions (33)

• theorem 1
• theorem 2: $L_\infty$-error bound with noise
• theorem 3: Bernstein
• theorem 4: Hanson-Wright
• corollary 1
• proof
• lemma 1: Matrix Chernoff
• proof
• lemma 2
• proof