Optimal kernel regression bounds under energy-bounded noise

TL;DR

The paper addresses the challenge of obtaining tight, non-asymptotic uncertainty bounds for kernel-based function estimation under energy-bounded (possibly correlated) noise. It introduces a relaxation with a test-point dependent noise kernel, connects the bound to Gaussian-process posterior mean and variance, and proves that the optimal bound is the infimum over a scalar noise parameter $\sigma$ of the relaxed bound. The main contributions include a closed-form relaxed bound $\overline{f}^{\sigma}(x_{N+1}) = f^{\mu}_{\sigma}(x_{N+1}) + \beta_{\sigma} \sqrt{\Sigma^{f}_{\sigma}(x_{N+1})}$ and an exact optimal bound $\overline{f}(x_{N+1}) = \inf_{\sigma>0} \overline{f}^{\sigma}(x_{N+1})$, with special cases recovering classical kernel-interpolation and linear-regression results. The work demonstrates practical impact through numerical comparisons and a safe-control example, showing the approach is competitive with probabilistic bounds and reliable when noise is correlated or biased.

Abstract

Non-conservative uncertainty bounds are key for both assessing an estimation algorithm's accuracy and in view of downstream tasks, such as its deployment in safety-critical contexts. In this paper, we derive a tight, non-asymptotic uncertainty bound for kernel-based estimation, which can also handle correlated noise sequences. Its computation relies on a mild norm-boundedness assumption on the unknown function and the noise, returning the worst-case function realization within the hypothesis class at an arbitrary query input location. The value of this function is shown to be given in terms of the posterior mean and covariance of a Gaussian process for an optimal choice of the measurement noise covariance. By rigorously analyzing the proposed approach and comparing it with other results in the literature, we show its effectiveness in returning tight and easy-to-compute bounds for kernel-based estimates.

Figures (3)

• Figure 1: Illustrative example for \ref{['thm:optimal_solution']}. The optimal upper and lower bounds (solid black) for the (unknown) latent function $f^{\mathrm{tr}}$ (dashed white) are determined by the relaxed bounds (shaded) around the GP posterior mean (dotted black) for an optimal choice of noise parameter $\sigma^\star_{\sup}$ (upper bound) and $\sigma^\star_{\inf}$ (lower bound). The three upper plots show the relaxed upper and lower bounds, $\overline{f}^\sigma$ and $\underline{f}^\sigma$ for the values $\sigma = \{ 10^{2}, 10^0, 10^{-2} \}$, respectively. The two bottom colorbars indicate the respective optimal values $\sigma^\star_{\sup}$ and $\sigma^\star_{\inf}$ for the upper and lower bound. The plotted relaxed upper (lower) bounds equal the optimal upper (lower) bound for each test point where the color of the shaded area matches the color indicated in the colorbar for the optimal value $\sigma^\star_{\sup}$ ($\sigma^\star_{\inf}$).
• Figure 2: Numerical comparison of area of uncertainty region for increasing number of data points $N$ with $\{5\%, 95\%\}$-percentiles shown in shade.
• Figure 3: Application of uncertainty bounds for safe control. Success rate (upper plot) and solve time (lower plot) with $\{ 5\%, 95\% \}$-percentiles shown in shade.

Theorems & Definitions (11)

• Lemma 1
• Theorem 1
• Proposition 1
• Proposition 2
• Corollary 1
• Lemma A.1
• proof
• Lemma A.2
• proof
• Corollary 2