Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Exploration Networks

Mattie Fellows, Brandon Kaplowitz, Christian Schroeder de Witt, Shimon Whiteson

TL;DR

A novel Bayesian model-free formulation and the first analysis showing that model-free approaches can yield Bayes-optimal policies are introduced, and empirical results demonstrate that BEN can learn true Bayes-optimal policies in tasks where existing model-free approaches fail.

Abstract

Bayesian reinforcement learning (RL) offers a principled and elegant approach for sequential decision making under uncertainty. Most notably, Bayesian agents do not face an exploration/exploitation dilemma, a major pathology of frequentist methods. However theoretical understanding of model-free approaches is lacking. In this paper, we introduce a novel Bayesian model-free formulation and the first analysis showing that model-free approaches can yield Bayes-optimal policies. We show all existing model-free approaches make approximations that yield policies that can be arbitrarily Bayes-suboptimal. As a first step towards model-free Bayes optimality, we introduce the Bayesian exploration network (BEN) which uses normalising flows to model both the aleatoric uncertainty (via density estimation) and epistemic uncertainty (via variational inference) in the Bellman operator. In the limit of complete optimisation, BEN learns true Bayes-optimal policies, but like in variational expectation-maximisation, partial optimisation renders our approach tractable. Empirical results demonstrate that BEN can learn true Bayes-optimal policies in tasks where existing model-free approaches fail.

Bayesian Exploration Networks

TL;DR

A novel Bayesian model-free formulation and the first analysis showing that model-free approaches can yield Bayes-optimal policies are introduced, and empirical results demonstrate that BEN can learn true Bayes-optimal policies in tasks where existing model-free approaches fail.

Abstract

Bayesian reinforcement learning (RL) offers a principled and elegant approach for sequential decision making under uncertainty. Most notably, Bayesian agents do not face an exploration/exploitation dilemma, a major pathology of frequentist methods. However theoretical understanding of model-free approaches is lacking. In this paper, we introduce a novel Bayesian model-free formulation and the first analysis showing that model-free approaches can yield Bayes-optimal policies. We show all existing model-free approaches make approximations that yield policies that can be arbitrarily Bayes-suboptimal. As a first step towards model-free Bayes optimality, we introduce the Bayesian exploration network (BEN) which uses normalising flows to model both the aleatoric uncertainty (via density estimation) and epistemic uncertainty (via variational inference) in the Bellman operator. In the limit of complete optimisation, BEN learns true Bayes-optimal policies, but like in variational expectation-maximisation, partial optimisation renders our approach tractable. Empirical results demonstrate that BEN can learn true Bayes-optimal policies in tasks where existing model-free approaches fail.
Paper Structure (42 sections, 8 theorems, 58 equations, 16 figures, 3 algorithms)

This paper contains 42 sections, 8 theorems, 58 equations, 16 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Contextual RL
  4. Bayesian RL
  5. Related Work
  6. Analysis of Model-Free BRL
  7. Model-Free BRL Formulation
  8. Aleatoric Uncertainty Matters
  9. Benefits of Model-Free over Model-Based BRL
  10. Myopic BRL
  11. QBRL
  12. Bayesian Exploration Network (ben)
  13. Network Specification
  14. Recurrent $Q$-Network
  15. Aleatoric Network
  16. ...and 27 more sections

Key Result

Theorem 3.2

Let ass:measurable hold, then $B^+[Q_\omega](h_t,a_t )=B^\star[Q_\omega](h_t,a_t )$.

Figures (16)

  • Figure 1: Sketch of transformation of variables $\beta_\omega$.
  • Figure 2: Evaluation in search and rescue episodic problem with no prior knowledge, showing cumulative return of ben vs RNN PPO and BootDQN+prior.
  • Figure 3: Evaluation in search and rescue episodic problem with weak prior knowledge, showing return of ben after each episode.
  • Figure 4: Evaluation in zero-shot search and rescue showing cumulative return for ben vs. QBRL methods.
  • Figure 5: Recurrent $Q$ Network
  • ...and 11 more figures

Theorems & Definitions (13)

  • Definition 2.1
  • Theorem 3.2
  • Corollary 3.4
  • Theorem 3.5
  • Corollary 3.6
  • Theorem 3.2
  • proof
  • Corollary 3.4
  • proof
  • Theorem 3.5
  • ...and 3 more