Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks

Jonathan Gornet, Yiannis Kantaros, Bruno Sinopoli

TL;DR

HyperController addresses the challenge of online hyperparameter optimization for reinforcement learning by modeling the hyperparameter objective as an unknown $LGDS$ and employing a learned Kalman-filter predictor to select hyperparameters during training. The approach reduces computational burden to $O(s^3)$ per update using a discretized, per-coordinate representation, and provides a regret bound to quantify performance. Empirically, it yields faster, more stable RL training and often higher rewards than GP-based Bayesian methods across OpenAI Gymnasium tasks. This framework offers a scalable alternative to Bayesian optimization for online hyperparameter control in reinforcement learning, with potential impact on faster deployment and iterative experimentation.

Abstract

We introduce Hyperparameter Controller (HyperController), a computationally efficient algorithm for hyperparameter optimization during training of reinforcement learning neural networks. HyperController optimizes hyperparameters quickly while also maintaining improvement of the reinforcement learning neural network, resulting in faster training and deployment. It achieves this by modeling the hyperparameter optimization problem as an unknown Linear Gaussian Dynamical System, which is a system with a state that linearly changes. It then learns an efficient representation of the hyperparameter objective function using the Kalman filter, which is the optimal one-step predictor for a Linear Gaussian Dynamical System. To demonstrate the performance of HyperController, it is applied as a hyperparameter optimizer during training of reinforcement learning neural networks on a variety of OpenAI Gymnasium environments. In four out of the five Gymnasium environments, HyperController achieves highest median reward during evaluation compared to other algorithms. The results exhibit the potential of HyperController for efficient and stable training of reinforcement learning neural networks.

HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks

TL;DR

HyperController addresses the challenge of online hyperparameter optimization for reinforcement learning by modeling the hyperparameter objective as an unknown and employing a learned Kalman-filter predictor to select hyperparameters during training. The approach reduces computational burden to per update using a discretized, per-coordinate representation, and provides a regret bound to quantify performance. Empirically, it yields faster, more stable RL training and often higher rewards than GP-based Bayesian methods across OpenAI Gymnasium tasks. This framework offers a scalable alternative to Bayesian optimization for online hyperparameter control in reinforcement learning, with potential impact on faster deployment and iterative experimentation.

Abstract

We introduce Hyperparameter Controller (HyperController), a computationally efficient algorithm for hyperparameter optimization during training of reinforcement learning neural networks. HyperController optimizes hyperparameters quickly while also maintaining improvement of the reinforcement learning neural network, resulting in faster training and deployment. It achieves this by modeling the hyperparameter optimization problem as an unknown Linear Gaussian Dynamical System, which is a system with a state that linearly changes. It then learns an efficient representation of the hyperparameter objective function using the Kalman filter, which is the optimal one-step predictor for a Linear Gaussian Dynamical System. To demonstrate the performance of HyperController, it is applied as a hyperparameter optimizer during training of reinforcement learning neural networks on a variety of OpenAI Gymnasium environments. In four out of the five Gymnasium environments, HyperController achieves highest median reward during evaluation compared to other algorithms. The results exhibit the potential of HyperController for efficient and stable training of reinforcement learning neural networks.
Paper Structure (19 sections, 8 theorems, 125 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 8 theorems, 125 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. A Representation for Predicting the Output of the Hyperparameter Cost Function
  4. Algorithm for Controller Hyperparameters During Neural Network Training
  5. Computational Efficiency
  6. Experiments
  7. Conclusion
  8. Derivation of the LGDS
  9. Discretization
  10. Modeling
  11. A Representation for Predicting the Output of the Hyperparameter Cost Function
  12. Alternating Directions
  13. Identifying the Representation
  14. Minor Details about the Numerical Results
  15. Proof for Theorem \ref{['theorem:continuity']}
  16. ...and 4 more sections

Key Result

Theorem 1

Let the reward $X_t$ be the output of the LGDS eq:LGDS based on equation eq:LGDS_output. Let $\hat{G}_{a}^t\left(\mathbf{c}_i\right)$ be learned accordingly to eq:learn_G. If actions are selected based on optimization problem eq:action_selection, then regret $R_n$ increases at the following rate wit

Figures (2)

  • Figure 1: The plots are the median training rewards (left plots) and median evaluation reward sum (right plots) for each method over are a function of wall clock time.
  • Figure 2: The plots are the median training rewards (left plots) and median evaluation reward sum (right plots) at the final training iteration.

Theorems & Definitions (15)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Theorem 3
  • proof
  • proof
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 5 more