Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation

TL;DR

Offline reinforcement learning suffers from extrapolation errors when learning from fixed offline data. MORAL replaces fixed horizon rollouts with an adversarial alternating-sampling framework over ensemble dynamics, augmented by a differential-factor regularization to curb extrapolation risk. The method yields superior policy learning and sample efficiency on the D4RL MuJoCo benchmarks, with strong robustness across tasks and reduced sensitivity to hyperparameters. This approach offers a scalable, horizon-free data augmentation strategy for practical offline RL in safety-critical domains.

Abstract

Model-based offline Reinforcement Learning (RL) constructs environment models from offline datasets to perform conservative policy optimization. Existing approaches focus on learning state transitions through ensemble models, rollouting conservative estimation to mitigate extrapolation errors. However, the static data makes it challenging to develop a robust policy, and offline agents cannot access the environment to gather new data. To address these challenges, we introduce Model-based Offline Reinforcement learning with AdversariaL data augmentation (MORAL). In MORAL, we replace the fixed horizon rollout by employing adversaria data augmentation to execute alternating sampling with ensemble models to enrich training data. Specifically, this adversarial process dynamically selects ensemble models against policy for biased sampling, mitigating the optimistic estimation of fixed models, thus robustly expanding the training data for policy optimization. Moreover, a differential factor is integrated into the adversarial process for regularization, ensuring error minimization in extrapolations. This data-augmented optimization adapts to diverse offline tasks without rollout horizon tuning, showing remarkable applicability. Extensive experiments on D4RL benchmark demonstrate that MORAL outperforms other model-based offline RL methods in terms of policy learning and sample efficiency.

Figures (10)

• Figure 1: (a) The framework of model-based offline RL. (b) We utilize a adversarial data augmentation process to execute alternating sampling, replacing the fixed horizon rollout marked red in (a). The primary player constructs candidate transition set and the second player selects biased samples to enrich training data without the rollout horizon configuration across datasets and tasks.
• Figure 2: The comparative experiments are conducted across various ensemble sizes and rollout horizons (h) within mixed datasets of two environments. We use stars (*) to denote the setups with optimal results. We find that the performance exhibited substantial differences in different settings.
• Figure 3: Overall framework of MORAL. (a) Initialize offline datasets $\mathcal{D}_{\rm{env}}$, $\mathcal{D}_{\rm{adv}}$ and train ensemble models. (b) Alternating sampling with two players for data augmentation in $\mathcal{D}_{\rm{adv}}$. (c) Update offline policy with both $\mathcal{D}_{\rm{env}}$ and $\mathcal{D}_{\rm{adv}}$, and provide sample policy $\pi^\theta$ in $\mathcal{D}_{\rm{adv}}$.
• Figure 4: Learning curves on walker2d-medium-expert and hopper-medium-expert datasets. Each number is the normalized reward during training, averaged over $12$ random seeds and the shadow is the standard error.
• Figure 5: Convergence step of PMDB and MORAL in $3$ expert tasks. The shaded regions are the standard deviation of each method.

Theorems & Definitions (4)

• Theorem 1
• proof
• Lemma 1
• proof