Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wiener Chaos in Kernel Regression: Towards Untangling Aleatoric and Epistemic Uncertainty

T. Faulwasser, O. Molodchyk

TL;DR

This approach allows us to distinguish the uncertainty that stems from the noise in the data samples from the total uncertainty encoded in the GP posterior distribution, and derive and discuss the analytic $\mathcal{L}^2$ solution to the arising Wiener kernel regression.

Abstract

Gaussian Processes (GPs) are a versatile method that enables different approaches towards learning for dynamics and control. Gaussianity assumptions appear in two dimensions in GPs: The positive semi-definite kernel of the underlying reproducing kernel Hilbert space is used to construct the co-variance of a Gaussian distribution over functions, while measurement noise (i.e. data corruption) is usually modeled as i.i.d. additive Gaussians. In this note, we generalize the setting and consider kernel ridge regression with additive i.i.d. non-Gaussian measurement noise. To apply the usual kernel trick, we rely on the representation of the uncertainty via polynomial chaos expansions, which are series expansions for random variables of finite variance introduced by Norbert Wiener. We derive and discuss the analytic $\mathcal{L}^2$ solution to the arising Wiener kernel regression. Considering a polynomial dynamic system as a numerical example, we show that our approach allows us to distinguish the uncertainty that stems from the noise in the data samples from the total uncertainty encoded in the GP posterior distribution.

Wiener Chaos in Kernel Regression: Towards Untangling Aleatoric and Epistemic Uncertainty

TL;DR

This approach allows us to distinguish the uncertainty that stems from the noise in the data samples from the total uncertainty encoded in the GP posterior distribution, and derive and discuss the analytic solution to the arising Wiener kernel regression.

Abstract

Gaussian Processes (GPs) are a versatile method that enables different approaches towards learning for dynamics and control. Gaussianity assumptions appear in two dimensions in GPs: The positive semi-definite kernel of the underlying reproducing kernel Hilbert space is used to construct the co-variance of a Gaussian distribution over functions, while measurement noise (i.e. data corruption) is usually modeled as i.i.d. additive Gaussians. In this note, we generalize the setting and consider kernel ridge regression with additive i.i.d. non-Gaussian measurement noise. To apply the usual kernel trick, we rely on the representation of the uncertainty via polynomial chaos expansions, which are series expansions for random variables of finite variance introduced by Norbert Wiener. We derive and discuss the analytic solution to the arising Wiener kernel regression. Considering a polynomial dynamic system as a numerical example, we show that our approach allows us to distinguish the uncertainty that stems from the noise in the data samples from the total uncertainty encoded in the GP posterior distribution.
Paper Structure (14 sections, 3 theorems, 22 equations, 2 figures)

This paper contains 14 sections, 3 theorems, 22 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Problem Statement and Preliminaries
  3. The weight-space view of GPs
  4. Problem statement
  5. Fundamentals of polynomial chaos expansions
  6. Main Results
  7. Regression with $\mathcal{L}^2$ noise description
  8. Wiener kernel regression
  9. Quantification of aleatoric and epistemic uncertainty
  10. Interpretation
  11. Numerical Example
  12. Gaussian measurement noise
  13. Non-Gaussian noise
  14. Conclusions and Outlook

Key Result

lemma thmcounterlemma

Suppose that the i.i.d. additive measurement noise $M_i\in \mathcal{L}^2(\Omega, \mathcal{F}, \mathbb{P}; \mathbb R), i \in \mathbb{I}_D$, admits an exact PCE of dimension $L=2$, i.e., $M_i = \mathsf{m}^{0} + \mathsf{m}^{1} \varphi^1(\xi_i)$. Then, the following statements hold:

Figures (2)

  • Figure 1: Comparison of the variance estimates from GP and Wiener kernel regression.
  • Figure 2: Example \ref{['eq:toy_problem']} with $\gamma$-distributed measurement noise.

Theorems & Definitions (7)

  • definition thmcounterdefinition: Exact PCE representation kit:muehlpfordt18a
  • lemma thmcounterlemma: Correspondence of optimization problems
  • proof
  • lemma thmcounterlemma: Moments of Wiener-kernel predictors
  • proof
  • lemma thmcounterlemma: Asymptotics of Wiener-kernel variance
  • proof