Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation
Amir Abolfazli, Zekun Song, Avishek Anand, Wolfgang Nejdl
TL;DR
The paper tackles offline RL under source–target domain mismatch by introducing Transition Scoring (TS) to quantify transition similarity to the target and Curriculum Learning-Based Trajectory Valuation (CLTV) to prioritize high-quality trajectories. TS leverages a learned transition scorer with a dynamics-discrepancy factor, while CLTV uses a dual-source/target actor–critic setup and a KL-divergence–based trajectory valuation to curate data. Theoretical results bound target-domain performance and show policy improvement guarantees, and extensive MuJoCo experiments on mixed D4RL datasets demonstrate that CLTV consistently surpasses baselines like Vanilla, CUORL, and Harness when combined with CQL or IQL. The work offers a practical, principled approach to leverage diverse datasets for offline RL with improved transferability and data efficiency, accompanied by code for reproducibility.
Abstract
The success of deep reinforcement learning (DRL) relies on the availability and quality of training data, often requiring extensive interactions with specific environments. In many real-world scenarios, where data collection is costly and risky, offline reinforcement learning (RL) offers a solution by utilizing data collected by domain experts and searching for a batch-constrained optimal policy. This approach is further augmented by incorporating external data sources, expanding the range and diversity of data collection possibilities. However, existing offline RL methods often struggle with challenges posed by non-matching data from these external sources. In this work, we specifically address the problem of source-target domain mismatch in scenarios involving mixed datasets, characterized by a predominance of source data generated from random or suboptimal policies and a limited amount of target data generated from higher-quality policies. To tackle this problem, we introduce Transition Scoring (TS), a novel method that assigns scores to transitions based on their similarity to the target domain, and propose Curriculum Learning-Based Trajectory Valuation (CLTV), which effectively leverages these transition scores to identify and prioritize high-quality trajectories through a curriculum learning approach. Our extensive experiments across various offline RL methods and MuJoCo environments, complemented by rigorous theoretical analysis, demonstrate that CLTV enhances the overall performance and transferability of policies learned by offline RL algorithms.