Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Data Exploitation in Deep Reinforcement Learning

Mingqi Yuan, Bo Li, Xin Jin, Wenjun Zeng

TL;DR

ADEPT addresses data efficiency and generalization in deep RL by adaptively scheduling the use of sampled data across learning stages via a multi-armed bandit formulation. It introduces three scheduling strategies—Upper Confidence Bound, Gaussian Thompson Sampling, and Round-Robin—to select update epochs (NUE) without adding auxiliary models, yielding substantial reductions in compute while improving performance on Procgen, MiniGrid, and PyBullet. The approach demonstrates strong data-efficiency gains and better generalization with plug-and-play applicability across on-policy RL methods like PPO and DrAC, suggesting practical impacts for scalable, energy-efficient RL training. By tightly coordinating data usage with learning progress, ADEPT enables more robust learning in diverse, data-constrained environments.

Abstract

We introduce ADEPT: Adaptive Data ExPloiTation, a simple yet powerful framework to enhance the **data efficiency** and **generalization** in deep reinforcement learning (RL). Specifically, ADEPT adaptively manages the use of sampled data across different learning stages via multi-armed bandit (MAB) algorithms, optimizing data utilization while mitigating overfitting. Moreover, ADEPT can significantly reduce the computational overhead and accelerate a wide range of RL algorithms. We test ADEPT on benchmarks including Procgen, MiniGrid, and PyBullet. Extensive simulation demonstrates that ADEPT can achieve superior performance with remarkable computational efficiency, offering a practical solution to data-efficient RL. Our code is available at https://github.com/yuanmingqi/ADEPT.

Adaptive Data Exploitation in Deep Reinforcement Learning

TL;DR

ADEPT addresses data efficiency and generalization in deep RL by adaptively scheduling the use of sampled data across learning stages via a multi-armed bandit formulation. It introduces three scheduling strategies—Upper Confidence Bound, Gaussian Thompson Sampling, and Round-Robin—to select update epochs (NUE) without adding auxiliary models, yielding substantial reductions in compute while improving performance on Procgen, MiniGrid, and PyBullet. The approach demonstrates strong data-efficiency gains and better generalization with plug-and-play applicability across on-policy RL methods like PPO and DrAC, suggesting practical impacts for scalable, energy-efficient RL training. By tightly coordinating data usage with learning progress, ADEPT enables more robust learning in diverse, data-constrained environments.

Abstract

We introduce ADEPT: Adaptive Data ExPloiTation, a simple yet powerful framework to enhance the **data efficiency** and **generalization** in deep reinforcement learning (RL). Specifically, ADEPT adaptively manages the use of sampled data across different learning stages via multi-armed bandit (MAB) algorithms, optimizing data utilization while mitigating overfitting. Moreover, ADEPT can significantly reduce the computational overhead and accelerate a wide range of RL algorithms. We test ADEPT on benchmarks including Procgen, MiniGrid, and PyBullet. Extensive simulation demonstrates that ADEPT can achieve superior performance with remarkable computational efficiency, offering a practical solution to data-efficient RL. Our code is available at https://github.com/yuanmingqi/ADEPT.
Paper Structure (50 sections, 19 equations, 37 figures, 3 tables, 1 algorithm)

This paper contains 50 sections, 19 equations, 37 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Efficiency in RL
  4. Generalization in RL
  5. MAB Algorithms for RL
  6. Background
  7. Reinforcement Learning
  8. Workflow of On-policy RL Algorithms
  9. Adaptive Data Exploitation
  10. Upper Confidence Bound
  11. Gaussian Thompson Sampling
  12. Round-Robin Scheduling
  13. Experiments
  14. Setup
  15. Results Analysis
  16. ...and 35 more sections

Figures (37)

  • Figure 1: Aggregated training performance and computational overhead comparison on the Procgen benchmark. ADEPT serves as a plug-and-play module to enhance RL algorithms, which can significantly promote data efficiency and reduce computational costs.
  • Figure 2: Overview of the ADEPT framework. (a) The proportion of the computational overhead (FLOPS) is evaluated using CleanRL's PPO implementation huang2022cleanrl and the Procgen benchmark cobbe2020leveraging. Since the overhead of the execution phase depends on various factors, it is not counted here. (b) A typical workflow of the on-policy RL algorithms. (c) ADEPT optimizes data utilization by adjusting the number of update epochs (NUE) in the update phase.
  • Figure 3: Training performance and computational overhead comparison of the PPO, DrAC, and their combinations with ADEPT on eight Procgen environments. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs. Note that the dotted line and dashed line represent the highest score and the lowest overhead, respectively.
  • Figure 4: Aggregated performance of the PPO, DrAC, and their combinations with ADEPT on the test levels of the Procgen benchmark. All the scores are normalized by the corresponding PPO scores, and bars indicate $95\%$ confidence intervals computed using stratified bootstrapping over five random seeds. Note that $*$ represents the best scores gathered from all three ADEPT algorithms.
  • Figure 5: The aggregated decision processes of ADEPT(U) and ADEPT(G) for PPO on the eight selected Procgen environments.
  • ...and 32 more figures