Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Reinforcement Learning: An Overview

Reza Refaei Afshar, Joaquin Vanschoren, Uzay Kaymak, Rui Zhang, Yaoxin Wu, Wen Song, Yingqian Zhang

TL;DR

The literature on automated RL is presented, including the recent large language model (LLM) based techniques, and the recent work on techniques that are not presently tailored for automated RL but hold promise for future integration into AutoRL are discussed.

Abstract

Reinforcement Learning and, recently, Deep Reinforcement Learning are popular methods for solving sequential decision-making problems modeled as Markov Decision Processes. RL modeling of a problem and selecting algorithms and hyper-parameters require careful consideration, as different configurations may entail completely different performances. These considerations are mainly the task of RL experts; however, RL is progressively becoming popular in other fields, such as combinatorial optimization, where researchers and system designers are not necessarily RL experts. Besides, many modeling decisions are typically made manually, such as defining state and action space, size of batches, batch update frequency, and time steps. For these reasons, automating different components of RL is of great importance, and it has attracted much attention in recent years. Automated RL provides a framework in which different components of RL, including MDP modeling, algorithm selection, and hyper-parameter optimization, are modeled and defined automatically. In this article, we present the literature on automated RL (AutoRL), including the recent large language model (LLM) based techniques. We also discuss the recent work on techniques that are not presently tailored for automated RL but hold promise for future integration into AutoRL. Furthermore, we discuss the challenges, open questions, and research directions in AutoRL.

Automated Reinforcement Learning: An Overview

TL;DR

The literature on automated RL is presented, including the recent large language model (LLM) based techniques, and the recent work on techniques that are not presently tailored for automated RL but hold promise for future integration into AutoRL are discussed.

Abstract

Reinforcement Learning and, recently, Deep Reinforcement Learning are popular methods for solving sequential decision-making problems modeled as Markov Decision Processes. RL modeling of a problem and selecting algorithms and hyper-parameters require careful consideration, as different configurations may entail completely different performances. These considerations are mainly the task of RL experts; however, RL is progressively becoming popular in other fields, such as combinatorial optimization, where researchers and system designers are not necessarily RL experts. Besides, many modeling decisions are typically made manually, such as defining state and action space, size of batches, batch update frequency, and time steps. For these reasons, automating different components of RL is of great importance, and it has attracted much attention in recent years. Automated RL provides a framework in which different components of RL, including MDP modeling, algorithm selection, and hyper-parameter optimization, are modeled and defined automatically. In this article, we present the literature on automated RL (AutoRL), including the recent large language model (LLM) based techniques. We also discuss the recent work on techniques that are not presently tailored for automated RL but hold promise for future integration into AutoRL. Furthermore, we discuss the challenges, open questions, and research directions in AutoRL.
Paper Structure (36 sections, 3 equations, 5 figures)

This paper contains 36 sections, 3 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Markov Decision Process Components
  3. Methods for Automating States
  4. Transferring raw observations to state representation
  5. Automatically defining the state representation
  6. Challenges
  7. Methods for Automating Actions
  8. Learning actions
  9. Discretizing continuous actions
  10. Challenges
  11. Automated Reward Function
  12. Curriculum Learning
  13. Bootstrapping
  14. Reward Shaping
  15. Challenges
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Solid arrows show the standard RL pipeline, while dashed arrows depict the AutoRL outer loop where evaluation feedback is used to update configurations and iterate until a satisfactory setting is found.
  • Figure 2: A roadmap of Automated Reinforcement Learning (AutoRL)
  • Figure 3: An example of Coarse coding. The resulting feature vector of state s is $(0,0,1,0,0,1)$.
  • Figure 4: An example of tile coding. The active tiles are shown in bold margins. The oval is the original observation space and the point is a sample observation.
  • Figure 5: The gradient descent using RNN