Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updates

TL;DR

The paper investigates when dynamics-invariant data augmentations improve data efficiency in model-free RL by introducing a controllable framework that isolates three DA properties: state-action coverage, reward density, and augmented replay ratio. Through experiments on PandaGym tasks and Goal2D, it finds that increasing state-action coverage typically yields larger gains than boosting reward density, and that reducing the augmented replay ratio (lower $\beta$) can dramatically improve learning and even enable solving tasks that are otherwise unsolvable. The results offer practical guidance: prioritize DA functions that expand coverage and carefully tune how augmented data is integrated, rather than simply generating more augmented data. This work advances understanding of when and why DA helps in RL and points to avenues for future exploration across architectures, hyperparameters, and more complex domains.

Abstract

Recently, data augmentation (DA) has emerged as a method for leveraging domain knowledge to inexpensively generate additional data in reinforcement learning (RL) tasks, often yielding substantial improvements in data efficiency. While prior work has demonstrated the utility of incorporating augmented data directly into model-free RL updates, it is not well-understood when a particular DA strategy will improve data efficiency. In this paper, we seek to identify general aspects of DA responsible for observed learning improvements. Our study focuses on sparse-reward tasks with dynamics-invariant data augmentation functions, serving as an initial step towards a more general understanding of DA and its integration into RL training. Experimentally, we isolate three relevant aspects of DA: state-action coverage, reward density, and the number of augmented transitions generated per update (the augmented replay ratio). From our experiments, we draw two conclusions: (1) increasing state-action coverage often has a much greater impact on data efficiency than increasing reward density, and (2) decreasing the augmented replay ratio substantially improves data efficiency. In fact, certain tasks in our empirical study are solvable only when the replay ratio is sufficiently low.

Figures (18)

• Figure 1: Visualizations of two augmentations -- translation (\ref{['fig:translate']}) and rotation (\ref{['fig:rotate']}) -- for a 2D navigation task in which an agent (black dot) must reach a goal (gold star). In \ref{['fig:training']}, "x$N$ policy data" corresponds to collecting $N$ times as many transitions with the agent's current policy between updates, and "x2 via rotate/translate" corresponds to generating one augmented transition per observed transition. We increase the batch size and replay buffer sizes proportionally to the amount of extra data to keep the replay ratio and replay age fixed across all experiments. We plot the interquartile mean success rate over 50 seeds with 95% bootstrap confidence belts.
• Figure 2: PandaPush-v3. A robotic arm must push a block to a goal location.
• Figure 3: "x2 policy data" agents double their learning data by collecting twice as many samples between updates, and "x2 data via TranslateGoal" agents double their learning data by generating one augmented transition per observed transition. Note that the horizontal axis shows the number of updates rather than timesteps. Each curve shows the interquartile mean over 10 seeds. Shaded regions denote 95% bootstrap confidence belts.
• Figure 4: Learning with TranslateGoal, TranslateGoalProximal(0), Translate, and TranslateProximal(0). We plot the IQM success rate over 10 seeds for Panda experiments and 50 for Goal2D. Shaded regions denote 95% bootstrap confidence belts.
• Figure 5: Learning with TranslateGoalProximal($p$) for various $p$. Darker colors correspond to larger $p$ values. The dashed red line for PickAndPlace denotes the final IQM success rate achieved by TranslateGoalProximal($0$) in Fig. \ref{['fig:coverage_prox']}. We plot the IQM success rate over 10 seeds for Panda experiments and 50 for Goal2D. Shaded regions denote 95% bootstrap confidence belts.

Theorems & Definitions (3)

• Definition 1
• Definition 2
• Definition 3