Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning

Abdul Wahab, Raksha Kumaraswamy, Martha White

TL;DR

This work introduces an algorithm for exploration called Value Bonuses with Ensemble errors (VBE), that maintains an ensemble of random action-value functions (RQFs) that uses the errors in the estimation of these RQFs to design value bonuses that provide first-visit optimism and deep exploration.

Abstract

Optimistic value estimates provide one mechanism for directed exploration in reinforcement learning (RL). The agent acts greedily with respect to an estimate of the value plus what can be seen as a value bonus. The value bonus can be learned by estimating a value function on reward bonuses, propagating local uncertainties around rewards. However, this approach only increases the value bonus for an action retroactively, after seeing a higher reward bonus from that state and action. Such an approach does not encourage the agent to visit a state and action for the first time. In this work, we introduce an algorithm for exploration called Value Bonuses with Ensemble errors (VBE), that maintains an ensemble of random action-value functions (RQFs). VBE uses the errors in the estimation of these RQFs to design value bonuses that provide first-visit optimism and deep exploration. The key idea is to design the rewards for these RQFs in such a way that the value bonus can decrease to zero. We show that VBE outperforms Bootstrap DQN and two reward bonus approaches (RND and ACB) on several classic environments used to test exploration and provide demonstrative experiments that it can scale easily to more complex environments like Atari.

Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning

TL;DR

This work introduces an algorithm for exploration called Value Bonuses with Ensemble errors (VBE), that maintains an ensemble of random action-value functions (RQFs) that uses the errors in the estimation of these RQFs to design value bonuses that provide first-visit optimism and deep exploration.

Abstract

Optimistic value estimates provide one mechanism for directed exploration in reinforcement learning (RL). The agent acts greedily with respect to an estimate of the value plus what can be seen as a value bonus. The value bonus can be learned by estimating a value function on reward bonuses, propagating local uncertainties around rewards. However, this approach only increases the value bonus for an action retroactively, after seeing a higher reward bonus from that state and action. Such an approach does not encourage the agent to visit a state and action for the first time. In this work, we introduce an algorithm for exploration called Value Bonuses with Ensemble errors (VBE), that maintains an ensemble of random action-value functions (RQFs). VBE uses the errors in the estimation of these RQFs to design value bonuses that provide first-visit optimism and deep exploration. The key idea is to design the rewards for these RQFs in such a way that the value bonus can decrease to zero. We show that VBE outperforms Bootstrap DQN and two reward bonus approaches (RND and ACB) on several classic environments used to test exploration and provide demonstrative experiments that it can scale easily to more complex environments like Atari.
Paper Structure (23 sections, 5 theorems, 17 equations, 14 figures, 3 tables, 1 algorithm)

This paper contains 23 sections, 5 theorems, 17 equations, 14 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Value Bonuses with Ensemble Errors
  4. Bonuses that reflect MDP-specific properties
  5. Using the Ensemble of Value Functions
  6. Guaranteeing Optimistic Initial Values
  7. Experiments
  8. Experimental Settings
  9. Pure Exploration
  10. Comparison in Classic Environments
  11. Atari
  12. Conclusion
  13. Experiment Details
  14. Environment Details
  15. Algorithm Details for Classics Control
  16. ...and 8 more sections

Key Result

Proposition 1

For all $i \in [k]$, we have $q^\pi_i = f_{i}$.

Figures (14)

  • Figure 1: Progression of unique states visited (grid size 50)
  • Figure 2: Contrasting the state coverage abilities of exploration algorithms in DeepSea. In (a) each bar corresponds to the total number of unique states visited by an agent after completing 10,000 episodes. The black stars indicate the total number of unique states for each grid size. Notably, VBE covers the entire state space, even for the larger grid sizes. (b) displays the progression of unique states visited by agents over the course of learning for Deepsea with grid size 50. The dotted line represents the total number of unique states (1275) in this environment. It provides evidence that VBE consistently explores new states at a significantly higher rate.
  • Figure 3: Online performance in River Swim, Puddle World, Mountain Car, and Deepsea. Higher on the y-axis is better. The x-axis denotes the number of interaction steps with the environment. The shaded region corresponds to standard errors.
  • Figure 4: Online performance in six Atari games, with shaded regions corresponding to standard errors. The x-axis is the number of environment interaction steps in millions, and the y-axis is the online Undiscounted Episodic Return, where higher is better. The environments in the second row are considered to be more challenging in terms of exploration.
  • Figure 5: Progression of unique states visited (grid size 50)
  • ...and 9 more figures

Theorems & Definitions (5)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Corollary 1: Corollary following from [Theorem 1]tagorti2015rate
  • Corollary 2: Corollary following from [Theorem 4.6]cai2019neural