Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset

TL;DR

This paper tackles the challenge of data efficiency in offline RL by introducing ReDOR, a method that frames dataset subset selection as a gradient-approximation problem and leverages a submodular reformulation of the actor-critic objective. It adapts Orthogonal Matching Pursuit to offline RL, incorporating stability enhancements such as empirical target costs, trajectory-based gradient matching, and returns-based data balancing. The authors provide convergence and approximation guarantees for the reduced dataset, and demonstrate that carefully selected smaller data subsets can match or surpass performance achieved with the full dataset across D4RL benchmarks while incurring significantly lower computational costs. The work suggests substantial practical impact for scalable offline RL, with potential extensions to real-world robotics tasks.

Abstract

Offline reinforcement learning (RL) represents a significant shift in RL research, allowing agents to learn from pre-collected datasets without further interaction with the environment. A key, yet underexplored, challenge in offline RL is selecting an optimal subset of the offline dataset that enhances both algorithm performance and training efficiency. Reducing dataset size can also reveal the minimal data requirements necessary for solving similar problems. In response to this challenge, we introduce ReDOR (Reduced Datasets for Offline RL), a method that frames dataset selection as a gradient approximation optimization problem. We demonstrate that the widely used actor-critic framework in RL can be reformulated as a submodular optimization objective, enabling efficient subset selection. To achieve this, we adapt orthogonal matching pursuit (OMP), incorporating several novel modifications tailored for offline RL. Our experimental results show that the data subsets identified by ReDOR not only boost algorithm performance but also do so with significantly lower computational complexity.

Figures (12)

• Figure 1: Experimental results on the D4RL (Hard) offline datasets. All experiment results were averaged over five random seeds. Our method achieves better or comparable results than the baselines with lower computational complexity.
• Figure 2: Experimental results on the D4RL offline datasets. All experiment results were averaged over five random seeds. Our method achieves better or comparable results than the baselines consistently.
• Figure 3: Visualization of the complete dataset and the reduced dataset in halfcheetah task. The higher opacity of a point represents a large time step towards the end of an episode. The dataset embedding is characterized by its division into different components. Samples selected by ReDOR connect different components by focusing on the data related to the task.
• Figure 4: Ablation results on D4RL (Hard) tasks with the normalized score metric.
• Figure 5: Visualization of selected data on hopper-medium-v0.

Theorems & Definitions (15)

• Theorem 4.1: Submodular Objective
• Theorem 5.1
• proof
• Theorem 5.2
• proof
• Corollary 5.3: Approximation Error Bound of the Reduced Dataset
• proof
• Theorem A.1: Submodular Objective
• proof
• Theorem A.1