Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning

Felix Pfeiffer, Shahram Eivazi

TL;DR

This paper proposes an enhancement to PBT by simultaneously utilizing both first- and second-order optimizers within a single population, and empirically demonstrate the potential of incorporating second-order optimizers within PBT-based RL.

Abstract

The tuning of hyperparameters in reinforcement learning (RL) is critical, as these parameters significantly impact an agent's performance and learning efficiency. Dynamic adjustment of hyperparameters during the training process can significantly enhance both the performance and stability of learning. Population-based training (PBT) provides a method to achieve this by continuously tuning hyperparameters throughout the training. This ongoing adjustment enables models to adapt to different learning stages, resulting in faster convergence and overall improved performance. In this paper, we propose an enhancement to PBT by simultaneously utilizing both first- and second-order optimizers within a single population. We conducted a series of experiments using the TD3 algorithm across various MuJoCo environments. Our results, for the first time, empirically demonstrate the potential of incorporating second-order optimizers within PBT-based RL. Specifically, the combination of the K-FAC optimizer with Adam led to up to a 10% improvement in overall performance compared to PBT using only Adam. Additionally, in environments where Adam occasionally fails, such as the Swimmer environment, the mixed population with K-FAC exhibited more reliable learning outcomes, offering a significant advantage in training stability without a substantial increase in computational time.

Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning

TL;DR

This paper proposes an enhancement to PBT by simultaneously utilizing both first- and second-order optimizers within a single population, and empirically demonstrate the potential of incorporating second-order optimizers within PBT-based RL.

Abstract

The tuning of hyperparameters in reinforcement learning (RL) is critical, as these parameters significantly impact an agent's performance and learning efficiency. Dynamic adjustment of hyperparameters during the training process can significantly enhance both the performance and stability of learning. Population-based training (PBT) provides a method to achieve this by continuously tuning hyperparameters throughout the training. This ongoing adjustment enables models to adapt to different learning stages, resulting in faster convergence and overall improved performance. In this paper, we propose an enhancement to PBT by simultaneously utilizing both first- and second-order optimizers within a single population. We conducted a series of experiments using the TD3 algorithm across various MuJoCo environments. Our results, for the first time, empirically demonstrate the potential of incorporating second-order optimizers within PBT-based RL. Specifically, the combination of the K-FAC optimizer with Adam led to up to a 10% improvement in overall performance compared to PBT using only Adam. Additionally, in environments where Adam occasionally fails, such as the Swimmer environment, the mixed population with K-FAC exhibited more reliable learning outcomes, offering a significant advantage in training stability without a substantial increase in computational time.
Paper Structure (16 sections, 12 equations, 5 figures, 7 tables)

This paper contains 16 sections, 12 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Second Order Optimization
  4. Dealing with Non-Convex Objective Functions: The Generalized Gauss-Newton Matrix
  5. Damping and the Trust-Region Problem
  6. Diagonal Gauss-Newton Second Order Optimizer
  7. Kronecker-Factored Approximate Curvature Optimizer
  8. Experiments
  9. Simulation Tasks
  10. Second-Order Optimizer
  11. Population Based Training
  12. Population Size
  13. Results
  14. First and Second-Order Optimizers in one Population
  15. Balancing Wall-Clock Runtime Across Optimizers
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Grid search for Diag. GGN across multiple environments, displaying mean rewards for various hyperparameter settings.
  • Figure 2: Grid search for K-FAC across multiple environments, displaying mean rewards for various hyperparameter settings.
  • Figure 3: Mean reward during training for single agents using different optimizer.
  • Figure 4: Training of Adam and Diag. GGN in one population.
  • Figure 5: Training of Adam and K-FAC in one population.