Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning

Siddharth Aravindan, Dixant Mittal, Wee Sun Lee

TL;DR

This work targets sample-efficient exploration in model-based reinforcement learning (MBRL) by leveraging Posterior Sampling through Thompson sampling-like posterior exploration. It introduces EVaDE, a trio of Gaussian dropout–based, event-aware convolutional layers designed for object-based domains to perturb object interactions, event importance, and spatial translations, thereby guiding exploration. By embedding EVaDE layers into the reward model of SimPLe, the authors approximate posterior sampling over rewards (PSRL-like behavior) and demonstrate that EVaDE-SimPLe achieves superior Atari 100K performance (mean HNS of 0.682, significantly beating baselines) with robust ablations showing complementary benefits across the three layers. The results suggest EVaDE provides a practical, architecture-friendly approach to exploration in object-centric MBRL, potentially complementing more search-based strategies in complex domains.

Abstract

Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function, which are intractable for tasks with high-dimensional state and action spaces. Recent works show that dropout, used in conjunction with neural networks, induces variational distributions that can approximate these posteriors. In this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), which are variational distributions that are useful for MBRL, especially when the underlying domain is object-based. We leverage the general domain knowledge of object-based domains to design three types of event-based convolutional layers to direct exploration. These layers rely on Gaussian dropouts and are inserted between the layers of the deep neural network model to help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE-equipped Simulated Policy Learning (EVaDE-SimPLe) on the 100K Atari game suite.

EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning

TL;DR

This work targets sample-efficient exploration in model-based reinforcement learning (MBRL) by leveraging Posterior Sampling through Thompson sampling-like posterior exploration. It introduces EVaDE, a trio of Gaussian dropout–based, event-aware convolutional layers designed for object-based domains to perturb object interactions, event importance, and spatial translations, thereby guiding exploration. By embedding EVaDE layers into the reward model of SimPLe, the authors approximate posterior sampling over rewards (PSRL-like behavior) and demonstrate that EVaDE-SimPLe achieves superior Atari 100K performance (mean HNS of 0.682, significantly beating baselines) with robust ablations showing complementary benefits across the three layers. The results suggest EVaDE provides a practical, architecture-friendly approach to exploration in object-centric MBRL, potentially complementing more search-based strategies in complex domains.

Abstract

Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function, which are intractable for tasks with high-dimensional state and action spaces. Recent works show that dropout, used in conjunction with neural networks, induces variational distributions that can approximate these posteriors. In this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), which are variational distributions that are useful for MBRL, especially when the underlying domain is object-based. We leverage the general domain knowledge of object-based domains to design three types of event-based convolutional layers to direct exploration. These layers rely on Gaussian dropouts and are inserted between the layers of the deep neural network model to help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE-equipped Simulated Policy Learning (EVaDE-SimPLe) on the 100K Atari game suite.
Paper Structure (34 sections, 1 theorem, 14 equations, 9 figures, 3 tables)

This paper contains 34 sections, 1 theorem, 14 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Event Based Variational Distributions
  4. Noisy Event Interaction Layer
  5. Noisy Event Weighting Layer
  6. Noisy Event Translation Layer
  7. Representational Capabilities of EVaDE networks
  8. Approximate PSRL with EVaDE equipped Simulated Policy Learning
  9. Experiments
  10. Network Architecture
  11. Experimental Details
  12. Results
  13. Ablation Studies
  14. Visualizations of the EVaDE layers
  15. Conclusion
  16. ...and 19 more sections

Key Result

theorem 1

Let $\mathbbm{n}$ be any neural network. For any convolutional layer $l$, let $m_i(l) \times n_i(l) \times c_i(l)$ and $m_o(l) \times n_o(l) \times c_o(l)$ denote the dimensions of its input and output respectively. Then, any function that can be represented by $\mathbbm{n}$ can also be represented

Figures (9)

  • Figure 1: Rewards in Breakout, a popular Atari game. (a) shows an interaction between the ball and a brick which gives the agent a positive reward. (b) shows a state, where the paddle is unable to prevent the ball from going out of bounds. The lack of this interaction between the agent and the ball in this situation results in a penalty for the agent.
  • Figure 2: (a) This image shows one noisy event interaction filter acting on an input with $c$ channels. Here $f$ is an $m \times m$ noisy convolutional filter, which acts upon input patches at the same location across different channels, noisily altering the value of events captured at those locations. (b) This image shows how the filters of the noisy event weighting layer weight the input channels. The filters $f_1, f_2, f_3$ and $f_c$ randomly upweight and downweight the events captured by the channels $c_1, c_2, c_3$ and $c_c$ respectively. The white entries of the filter are entries that are set to zero, while the rest are trainable noisy model parameters. (c) The noisy event translation filter. The filters $f_1, f_2, f_3$ and $f_c$ noisily translate events/objects captured by the channels $c_1, c_2, c_3$ and $c_c$ respectively. The white entries of the filter are entries that are set to zero, while the rest are trainable noisy model parameters. Gaussian multiplicative dropout is applied to all the non-zero parameters of all EVaDE filters.
  • Figure 3: The network architecture of the environment model used to train EVaDE-SimPLe.
  • Figure 4: Learning curves of EVaDE-SimPLe agents, SimPLe(30) agents and agents which only add one of the EVaDE layers trained on the 12 game subset of the Atari 100K test suite.
  • Figure 5: This figure shows the output map that captures interactions between two input maps when passed through the noisy event interaction layer.
  • ...and 4 more figures

Theorems & Definitions (7)

  • theorem 1
  • Claim 1
  • proof
  • Claim 2
  • proof
  • Claim 3
  • proof