Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Training Mechanism for Reinforcement Learning via Online Model Selection

Aida Afshar, Aldo Pacchiano

TL;DR

The paper tackles the mismatch between RL theory and practice by introducing online model selection to dynamically allocate training compute across a set of base agents. It formulates a data-driven selector framework, notably the Doubling Data Driven Regret Balancing (D^3RB) algorithm, and proves regret bounds while integrating the selector into general RL training. Empirically, it demonstrates near-oracle performance in neural architecture selection, robustness to non-stationary dynamics in step-size tuning, and stabilization via seed diversity. The work highlights the practical value of data-driven, algorithm-agnostic model selection for improving RL efficiency and reliability.

Abstract

We study the problem of online model selection in reinforcement learning, where the selector has access to a class of reinforcement learning agents and learns to adaptively select the agent with the right configuration. Our goal is to establish the improved efficiency and performance gains achieved by integrating online model selection methods into reinforcement learning training procedures. We examine the theoretical characterizations that are effective for identifying the right configuration in practice, and address three practical criteria from a theoretical perspective: 1) Efficient resource allocation, 2) Adaptation under non-stationary dynamics, and 3) Training stability across different seeds. Our theoretical results are accompanied by empirical evidence from various model selection tasks in reinforcement learning, including neural architecture selection, step-size selection, and self model selection.

Improved Training Mechanism for Reinforcement Learning via Online Model Selection

TL;DR

The paper tackles the mismatch between RL theory and practice by introducing online model selection to dynamically allocate training compute across a set of base agents. It formulates a data-driven selector framework, notably the Doubling Data Driven Regret Balancing (D^3RB) algorithm, and proves regret bounds while integrating the selector into general RL training. Empirically, it demonstrates near-oracle performance in neural architecture selection, robustness to non-stationary dynamics in step-size tuning, and stabilization via seed diversity. The work highlights the practical value of data-driven, algorithm-agnostic model selection for improving RL efficiency and reliability.

Abstract

We study the problem of online model selection in reinforcement learning, where the selector has access to a class of reinforcement learning agents and learns to adaptively select the agent with the right configuration. Our goal is to establish the improved efficiency and performance gains achieved by integrating online model selection methods into reinforcement learning training procedures. We examine the theoretical characterizations that are effective for identifying the right configuration in practice, and address three practical criteria from a theoretical perspective: 1) Efficient resource allocation, 2) Adaptation under non-stationary dynamics, and 3) Training stability across different seeds. Our theoretical results are accompanied by empirical evidence from various model selection tasks in reinforcement learning, including neural architecture selection, step-size selection, and self model selection.

Paper Structure

This paper contains 25 sections, 8 theorems, 44 equations, 10 figures, 1 table, 7 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement Learning
  4. Model Selection
  5. Training Mechanism
  6. Algorithm
  7. Model Selection Tasks
  8. Neural Architecture Selection
  9. Step-Size Selection
  10. Bandit Algorithms for Online Model Selection: What would go wrong?
  11. Self-Model Selection
  12. Discussion and Future Work
  13. Conclusion
  14. Theoretical Results
  15. Misspecification Test
  16. ...and 10 more sections

Key Result

Theorem 2

Denote the event $\mathcal{E}$, for a parameter $\delta$ and universal constant $c$. Under event $\mathcal{E}$, the total regret incurred by data-driven regret balancing (D$^3$RB) after $T$ rounds satisfies, With probability $1-\delta$.

Figures (10)

  • Figure 1: Selector (D$^3$RB)
  • Figure 2: Neural Architecture Selection for DQN algorithm in Atari Environments. Comparison of D$^3$RB with individual runs of base agents shows that the selector has comparable performance to the best solo base agent. Curves show the average and standard deviation over three seeds.
  • Figure 3: Q-Network architecture for three base agents in neural architecture selection task
  • Figure 4: Comparison of 6 model selection strategies in the step-size selection task for the PPO algorithm. Each curve shows the average and standard deviation over three seeds.
  • Figure 5: D$^3$RB Selection Statistics in MuJoco Environments
  • ...and 5 more figures

Theorems & Definitions (9)

  • Definition 1: Regret Coefficient
  • Theorem 2: dann2024data
  • Proposition 3: Misspecification Test
  • Theorem 4
  • Theorem 5
  • Lemma 6
  • Lemma 7
  • Theorem 8
  • Proposition 9