Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural-Kernel Conditional Mean Embeddings

Eiki Shimizu, Kenji Fukumizu, Dino Sejdinovic

TL;DR

This work proposes a new method that effectively combines the strengths of deep learning with CMEs in order to address challenges of scalability and expressiveness challenges, and leverages the end-to-end neural network optimization framework using a kernel-based objective.

Abstract

Kernel conditional mean embeddings (CMEs) offer a powerful framework for representing conditional distribution, but they often face scalability and expressiveness challenges. In this work, we propose a new method that effectively combines the strengths of deep learning with CMEs in order to address these challenges. Specifically, our approach leverages the end-to-end neural network (NN) optimization framework using a kernel-based objective. This design circumvents the computationally expensive Gram matrix inversion required by current CME methods. To further enhance performance, we provide efficient strategies to optimize the remaining kernel hyperparameters. In conditional density estimation tasks, our NN-CME hybrid achieves competitive performance and often surpasses existing deep learning-based methods. Lastly, we showcase its remarkable versatility by seamlessly integrating it into reinforcement learning (RL) contexts. Building on Q-learning, our approach naturally leads to a new variant of distributional RL methods, which demonstrates consistent effectiveness across different environments.

Neural-Kernel Conditional Mean Embeddings

TL;DR

This work proposes a new method that effectively combines the strengths of deep learning with CMEs in order to address challenges of scalability and expressiveness challenges, and leverages the end-to-end neural network optimization framework using a kernel-based objective.

Abstract

Kernel conditional mean embeddings (CMEs) offer a powerful framework for representing conditional distribution, but they often face scalability and expressiveness challenges. In this work, we propose a new method that effectively combines the strengths of deep learning with CMEs in order to address these challenges. Specifically, our approach leverages the end-to-end neural network (NN) optimization framework using a kernel-based objective. This design circumvents the computationally expensive Gram matrix inversion required by current CME methods. To further enhance performance, we provide efficient strategies to optimize the remaining kernel hyperparameters. In conditional density estimation tasks, our NN-CME hybrid achieves competitive performance and often surpasses existing deep learning-based methods. Lastly, we showcase its remarkable versatility by seamlessly integrating it into reinforcement learning (RL) contexts. Building on Q-learning, our approach naturally leads to a new variant of distributional RL methods, which demonstrates consistent effectiveness across different environments.
Paper Structure (34 sections, 1 theorem, 42 equations, 3 figures, 7 tables)

This paper contains 34 sections, 1 theorem, 42 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Kernel Mean Embeddings
  4. Kernel Conditional Mean Embeddings
  5. Deep Feature Approach
  6. Neural Network Based CME
  7. Model and Objective Function
  8. Choice of Kernel $k_\mathcal{Y}$
  9. Approach 1: Iterative Optimization
  10. Approach 2: Joint Optimization
  11. Related Work
  12. Experiments on Density Estimation
  13. Toy Data
  14. UCI Datasets
  15. Applications to Reinforcement Learning
  16. ...and 19 more sections

Key Result

Theorem 3.1

Let $f= \sum_{a=1}^{M} k_{\sqrt{2}\sigma}(\cdot, \eta_{a})w_a\in\mathcal{H}_{\sqrt{2}\sigma}$ and $g=\sum_{a=1}^{M} k_{\sigma}(\cdot, \eta_{a})w_a\in\mathcal{H}_{\sigma}$. Then the following inequality holds:

Figures (3)

  • Figure 1: Performance comparison on three environments. We report the mean of cumulative rewards across 10 independent runs.
  • Figure 2: Visualization of toy datasets
  • Figure 3: Comparison of our proposed methods using single and fused kernels. We report the mean of cumulative rewards across 10 independent runs.

Theorems & Definitions (1)

  • Theorem 3.1