Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentForge: A Flexible Low-Code Platform for Reinforcement Learning Agent Design

Francisco Erivaldo Fernandes Junior, Antti Oulasvirta

TL;DR

AgentForge is presented, a flexible low-code platform to optimize any parameter set across an RL system, which allows an optimization problem to be defined in a few lines of code and handed to any of the interfaced optimizers.

Abstract

Developing a reinforcement learning (RL) agent often involves identifying values for numerous parameters, covering the policy, reward function, environment, and agent-internal architecture. Since these parameters are interrelated in complex ways, optimizing them is a black-box problem that proves especially challenging for nonexperts. Although existing optimization-as-a-service platforms (e.g., Vizier and Optuna) can handle such problems, they are impractical for RL systems, since the need for manual user mapping of each parameter to distinct components makes the effort cumbersome. It also requires understanding of the optimization process, limiting the systems' application beyond the machine learning field and restricting access in areas such as cognitive science, which models human decision-making. To tackle these challenges, the paper presents AgentForge, a flexible low-code platform to optimize any parameter set across an RL system. Available at https://github.com/feferna/AgentForge, it allows an optimization problem to be defined in a few lines of code and handed to any of the interfaced optimizers. With AgentForge, the user can optimize the parameters either individually or jointly. The paper presents an evaluation of its performance for a challenging vision-based RL problem.

AgentForge: A Flexible Low-Code Platform for Reinforcement Learning Agent Design

TL;DR

AgentForge is presented, a flexible low-code platform to optimize any parameter set across an RL system, which allows an optimization problem to be defined in a few lines of code and handed to any of the interfaced optimizers.

Abstract

Developing a reinforcement learning (RL) agent often involves identifying values for numerous parameters, covering the policy, reward function, environment, and agent-internal architecture. Since these parameters are interrelated in complex ways, optimizing them is a black-box problem that proves especially challenging for nonexperts. Although existing optimization-as-a-service platforms (e.g., Vizier and Optuna) can handle such problems, they are impractical for RL systems, since the need for manual user mapping of each parameter to distinct components makes the effort cumbersome. It also requires understanding of the optimization process, limiting the systems' application beyond the machine learning field and restricting access in areas such as cognitive science, which models human decision-making. To tackle these challenges, the paper presents AgentForge, a flexible low-code platform to optimize any parameter set across an RL system. Available at https://github.com/feferna/AgentForge, it allows an optimization problem to be defined in a few lines of code and handed to any of the interfaced optimizers. With AgentForge, the user can optimize the parameters either individually or jointly. The paper presents an evaluation of its performance for a challenging vision-based RL problem.

Paper Structure

This paper contains 18 sections, 3 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. THE PROPOSED FRAMEWORK
  4. General Characterization
  5. Specifying the Optimization Task
  6. Optimization Algorithms
  7. Parallel Evaluation of Trials
  8. EVALUATION
  9. A Pixel-Based POMDP Agent
  10. Optimization Configuration
  11. The Parameters Optimized
  12. Joint vs. Individual-Parameter Optimization
  13. RESULTS
  14. Results from Joint Optimization
  15. Results of the Joint vs. Parameter-Wise Optimization
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The AgentForge platform in overview. a) The inputs consist of three elements: optimization settings (including search parameters), a customized environment, and an evaluation method. b) AgentForge executes joint optimization of RL parameters through a two-loop process. c) Then, AgentForge provides the user with the best-performing agent along with the optimization results and performance statistics.
  • Figure 2: Mean reward per episode for the pixel-based Lunar Lander agent under different optimization strategies. Top: Joint optimization of parameters and NAS using random search, BayesOpt, and PSO. Bottom: A plot of joint vs. individual-parameter optimization using BayesOpt, with and without NAS, and NAS-only optimization.
  • Figure : Sample code snippet illustrating parameter definition, creation of the environment, and evaluation setup.