Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Exploration Methods in Reinforcement Learning

Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, Doina Precup

TL;DR

The paper surveys exploration methods for sequential decision making in reinforcement learning, organizing approaches into reward-free versus reward-based categories and further distinguishing memory-free from memory-based strategies. It covers early random and intrinsic-motivation techniques, randomized action selection (value-based and policy-search), optimism/bonus-based methods (tabular and function-approximation), prediction-error bonuses, and deliberate Bayesian/meta-learning frameworks, including PSRL and BAMCP. The work highlights theoretical guarantees (PAC, UCB, BAMDP), practical performance, and the challenges of evaluating exploration methods across diverse tasks and benchmarks. By compiling a comprehensive taxonomy and representative methods, it provides practitioners with a structured entry point to select exploration strategies aligned with problem structure and computational resources.

Abstract

Exploration is an essential component of reinforcement learning algorithms, where agents need to learn how to predict and control unknown and often stochastic environments. Reinforcement learning agents depend crucially on exploration to obtain informative data for the learning process as the lack of enough information could hinder effective learning. In this article, we provide a survey of modern exploration methods in (Sequential) reinforcement learning, as well as a taxonomy of exploration methods.

A Survey of Exploration Methods in Reinforcement Learning

TL;DR

The paper surveys exploration methods for sequential decision making in reinforcement learning, organizing approaches into reward-free versus reward-based categories and further distinguishing memory-free from memory-based strategies. It covers early random and intrinsic-motivation techniques, randomized action selection (value-based and policy-search), optimism/bonus-based methods (tabular and function-approximation), prediction-error bonuses, and deliberate Bayesian/meta-learning frameworks, including PSRL and BAMCP. The work highlights theoretical guarantees (PAC, UCB, BAMDP), practical performance, and the challenges of evaluating exploration methods across diverse tasks and benchmarks. By compiling a comprehensive taxonomy and representative methods, it provides practitioners with a structured entry point to select exploration strategies aligned with problem structure and computational resources.

Abstract

Exploration is an essential component of reinforcement learning algorithms, where agents need to learn how to predict and control unknown and often stochastic environments. Reinforcement learning agents depend crucially on exploration to obtain informative data for the learning process as the lack of enough information could hinder effective learning. In this article, we provide a survey of modern exploration methods in (Sequential) reinforcement learning, as well as a taxonomy of exploration methods.

Paper Structure

This paper contains 38 sections, 34 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Notation and Background
  3. Markov Decision Processes
  4. Reinforcement Learning Setting
  5. Categorization of Exploratory Techniques
  6. Reward-Free Exploration
  7. Blind Exploration
  8. Intrinsically-Motivated Exploration
  9. Randomized Action Selection
  10. Value-Based Methods
  11. Policy-Search Based Methods
  12. Perturbed space and coherence
  13. Action-space perturbing strategies
  14. Parameter-space perturbing strategies
  15. The distribution of perturbations
  16. ...and 23 more sections

Figures (2)

  • Figure 1: Exploration Categories- The exploration methods are categorized into two main groups reward-free and reward-based exploration techniques, depending on their utilization of extrinsic rewards. Each group is further divided to memory-based and memory-free categories based on the reliance of the exploratory decisions on the agent's memory of the observed space.
  • Figure 2: A 2-state MDP with uncertain transition probabilities under (a) action 1 and (b) action 2. Rewards are denoted by $\pm 1$ in the states.