Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Reinforcement Learning Through Guided Search

Jérôme Arjonilla, Abdallah Saffidine, Tristan Cazenave

TL;DR

MCTS renowned for its state-of-the-art capabilities across various domains, catches the interest due to its ability to converge to equilibrium in single-player and two-player contexts, and is harnessed as a guide for the RL agent.

Abstract

With the aim of improving performance in Markov Decision Problem in an Off-Policy setting, we suggest taking inspiration from what is done in Offline Reinforcement Learning (RL). In Offline RL, it is a common practice during policy learning to maintain proximity to a reference policy to mitigate uncertainty, reduce potential policy errors, and help improve performance. We find ourselves in a different setting, yet it raises questions about whether a similar concept can be applied to enhance performance ie, whether it is possible to find a guiding policy capable of contributing to performance improvement, and how to incorporate it into our RL agent. Our attention is particularly focused on algorithms based on Monte Carlo Tree Search (MCTS) as a guide.MCTS renowned for its state-of-the-art capabilities across various domains, catches our interest due to its ability to converge to equilibrium in single-player and two-player contexts. By harnessing the power of MCTS as a guide for our RL agent, we observed a significant performance improvement, surpassing the outcomes achieved by utilizing each method in isolation. Our experiments were carried out on the Atari 100k benchmark.

Enhancing Reinforcement Learning Through Guided Search

TL;DR

MCTS renowned for its state-of-the-art capabilities across various domains, catches the interest due to its ability to converge to equilibrium in single-player and two-player contexts, and is harnessed as a guide for the RL agent.

Abstract

With the aim of improving performance in Markov Decision Problem in an Off-Policy setting, we suggest taking inspiration from what is done in Offline Reinforcement Learning (RL). In Offline RL, it is a common practice during policy learning to maintain proximity to a reference policy to mitigate uncertainty, reduce potential policy errors, and help improve performance. We find ourselves in a different setting, yet it raises questions about whether a similar concept can be applied to enhance performance ie, whether it is possible to find a guiding policy capable of contributing to performance improvement, and how to incorporate it into our RL agent. Our attention is particularly focused on algorithms based on Monte Carlo Tree Search (MCTS) as a guide.MCTS renowned for its state-of-the-art capabilities across various domains, catches our interest due to its ability to converge to equilibrium in single-player and two-player contexts. By harnessing the power of MCTS as a guide for our RL agent, we observed a significant performance improvement, surpassing the outcomes achieved by utilizing each method in isolation. Our experiments were carried out on the Atari 100k benchmark.
Paper Structure (47 sections, 22 equations, 9 figures, 6 tables)

This paper contains 47 sections, 22 equations, 9 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Formalism and Notation
  3. Markov Decision Process
  4. Reinforcement Learning
  5. Critic
  6. Actor
  7. Behavior Cloning
  8. Search Algorithm
  9. Monte Carlo Tree Search
  10. Guide
  11. Analysis of the various guides
  12. Human
  13. Random
  14. Behavior Cloning
  15. Search Algorithm
  16. ...and 32 more sections

Figures (9)

  • Figure 1: Aggregate performance. Shaded area shows $95\%$ stratified bootstrap confidence interval. The x-axis represents the human normalized score.
  • Figure 2: Learning curves on $2$ different game of Atari100k benchmarks with $3$ algorithms presented. The shaded area shows $95\%$ confidence interval.
  • Figure 3: Percentage improvement of algorithm X compared to algorithm Y on Atari100k Benchmarks. Improvement is measured as a percentage of mean human-normalized return.
  • Figure 4: Aggregate performance metrics according to the weight. The shaded area shows $95\%$ stratified bootstrap confidence interval. The x-axis represents the human normalized score.
  • Figure 5: Aggregate performance according to the number of calls made to the guide. The shaded area shows $95\%$ stratified bootstrap confidence interval. The x-axis represents the human normalised score.
  • ...and 4 more figures