Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recursive Estimation of Conditional Kernel Mean Embeddings

Ambrus Tamás, Balázs Csanád Csáji

TL;DR

This paper presents a new recursive algorithm to estimate the conditional kernel mean map in a Hilbert space valued $L_2$ space, that is in a Bochner space, and proves the weak and strong consistency of the recursive estimator under mild conditions.

Abstract

Kernel mean embeddings, a widely used technique in machine learning, map probability distributions to elements of a reproducing kernel Hilbert space (RKHS). For supervised learning problems, where input-output pairs are observed, the conditional distribution of outputs given the inputs is a key object. The input dependent conditional distribution of an output can be encoded with an RKHS valued function, the conditional kernel mean map. In this paper we present a new recursive algorithm to estimate the conditional kernel mean map in a Hilbert space valued $L_2$ space, that is in a Bochner space. We prove the weak and strong $L_2$ consistency of our recursive estimator under mild conditions. The idea is to generalize Stone's theorem for Hilbert space valued regression in a locally compact Polish space. We present new insights about conditional kernel mean embeddings and give strong asymptotic bounds regarding the convergence of the proposed recursive method. Finally, the results are demonstrated on three application domains: for inputs coming from Euclidean spaces, Riemannian manifolds and locally compact subsets of function spaces.

Recursive Estimation of Conditional Kernel Mean Embeddings

TL;DR

This paper presents a new recursive algorithm to estimate the conditional kernel mean map in a Hilbert space valued space, that is in a Bochner space, and proves the weak and strong consistency of the recursive estimator under mild conditions.

Abstract

Kernel mean embeddings, a widely used technique in machine learning, map probability distributions to elements of a reproducing kernel Hilbert space (RKHS). For supervised learning problems, where input-output pairs are observed, the conditional distribution of outputs given the inputs is a key object. The input dependent conditional distribution of an output can be encoded with an RKHS valued function, the conditional kernel mean map. In this paper we present a new recursive algorithm to estimate the conditional kernel mean map in a Hilbert space valued space, that is in a Bochner space. We prove the weak and strong consistency of our recursive estimator under mild conditions. The idea is to generalize Stone's theorem for Hilbert space valued regression in a locally compact Polish space. We present new insights about conditional kernel mean embeddings and give strong asymptotic bounds regarding the convergence of the proposed recursive method. Finally, the results are demonstrated on three application domains: for inputs coming from Euclidean spaces, Riemannian manifolds and locally compact subsets of function spaces.
Paper Structure (20 sections, 27 theorems, 120 equations)

This paper contains 20 sections, 27 theorems, 120 equations.

Table of Contents

  1. Introduction
  2. Related works
  3. Contributions
  4. Applications
  5. Kernel Mean Embeddings of Distributions
  6. Statistical Framework
  7. Reproducing Kernel Hilbert Spaces
  8. Kernel Mean Embeddings of Marginal Distributions
  9. Recursive Estimation of Kernel Mean Embeddings
  10. Conditional Kernel Mean Embeddings
  11. Recovering the regression function
  12. Recovering a family of real-valued regression
  13. Universal Consistency
  14. Generalization of Stone's Theorem
  15. Recursive Estimation of Conditional Kernel Mean Embeddings
  16. ...and 5 more sections

Key Result

Theorem 1

Let $\{Y_i\}_{i=1}^n$ be an i.i.d. sample from distribution $Q_{{Y}}$ on a separable topological space $\mathbb{Y}$. Assume that $\ell(\cdot,y)$ is continuous and there exists $C_\ell > 0$ such that $\ell(y,y) \leq C_\ell$ for all $y \in \mathbb{Y}$. Then, for all $\delta \in (0,1)$ with probability

Theorems & Definitions (32)

  • Theorem 1
  • Theorem 2
  • Lemma 3
  • Lemma 4
  • Definition 5
  • Theorem 6
  • Lemma 7
  • Lemma 8
  • Definition 9: smoother
  • Theorem 10
  • ...and 22 more