Augmenting Offline RL with Unlabeled Data

TL;DR

This work tackles the OOD issue in offline RL by moving beyond conservative, data-only strategies. It introduces Ludor, a teacher–student framework where a teacher trained on unlabeled data via behavior cloning informs a student learning from an offline RL dataset through EMA-based knowledge transfer, with a cosine-based policy discrepancy measure ($\kappa$) to weight losses by data reliability. The method integrates with existing actor–critic offline RL algorithms (e.g., TD3BC, IQL) and demonstrates improved performance across MuJoCo tasks and AntMaze, while also showing that the student can outperform the teacher in many tasks. Key contributions include the empirical validation of using unlabeled data for knowledge transfer in offline RL, the introduction of a non-probabilistic policy discrepancy score to mitigate extrapolation errors, and comprehensive ablations and robustness analyses across data coverage, removal ratios, and observation-space perturbations. Overall, Ludor provides a practical mechanism to leverage domain-knowledge-rich unlabeled data to alleviate OOD pitfalls without requiring full state-action coverage, broadening the applicability of offline RL in real-world settings.

Abstract

Recent advancements in offline Reinforcement Learning (Offline RL) have led to an increased focus on methods based on conservative policy updates to address the Out-of-Distribution (OOD) issue. These methods typically involve adding behavior regularization or modifying the critic learning objective, focusing primarily on states or actions with substantial dataset support. However, we challenge this prevailing notion by asserting that the absence of an action or state from a dataset does not necessarily imply its suboptimality. In this paper, we propose a novel approach to tackle the OOD problem. We introduce an offline RL teacher-student framework, complemented by a policy similarity measure. This framework enables the student policy to gain insights not only from the offline RL dataset but also from the knowledge transferred by a teacher policy. The teacher policy is trained using another dataset consisting of state-action pairs, which can be viewed as practical domain knowledge acquired without direct interaction with the environment. We believe this additional knowledge is key to effectively solving the OOD issue. This research represents a significant advancement in integrating a teacher-student network into the actor-critic framework, opening new avenues for studies on knowledge transfer in offline RL and effectively addressing the OOD challenge.

Figures (17)

• Figure 1: Ludor consists of two networks of the same architecture but difference weight, named the teacher network and the student network respectively. The teacher $T_{\sigma}$ learns the knowledge via $\mathcal{D}_d$ by using BC, while the student learns from OOD data $\mathcal{D}$, and augmented by the knowledge transfered from the teacher via EMA. At the same time, the discrepancy measures are computed, and assign a vector weight to the loss function of actor-critic training.
• Figure 2: Average normalized score on comparative analysis of Ludor built on TD3BC and baseline models across four environments: HalfCheetah, Hopper, Walker2D, and Antmaze, presented sequentially from left to right. The unlabeled data used is expert data.
• Figure 3: Average normalized score on comparative analysis of Ludor built on IQL and baseline models across four environments: HalfCheetah, Hopper, Walker2D, and Antmaze, presented sequentially from left to right. The unlabeled data used is expert data.
• Figure 4: Average normalized score on comparative analysis of Ludor built on TD3BC and baseline models across four environments: HalfCheetah, Hopper, Walker2D, and Antmaze, presented sequentially from left to right. The unlabeled data used is medium data.
• Figure 5: Average normalized score on comparative analysis of Ludor built on IQL and baseline models across four environments: HalfCheetah, Hopper, Walker2D, and Antmaze, presented sequentially from left to right. The unlabeled data used is medium data.