Sample-Efficient Policy Constraint Offline Deep Reinforcement Learning based on Sample Filtering

TL;DR

This work tackles distribution shift in offline RL by introducing a simple sample-filtering strategy that scores episodes via average and discounted rewards to curate high-quality trajectories. The filtered dataset is used to train policy-constraint offline RL methods, yielding improved sample efficiency and stronger final performance across multiple benchmarks. Experimental results on BEAR, TD3+BC, and IQL with D4RL datasets demonstrate outperformance of baselines in most tasks and exhibit faster convergence due to focused learning on high-value regions. The approach offers a practical pre-processing step to enhance offline RL pipelines, with potential extensions to richer filtering criteria and real-world applications.

Abstract

Offline reinforcement learning (RL) aims to learn a policy that maximizes the expected return using a given static dataset of transitions. However, offline RL faces the distribution shift problem. The policy constraint offline RL method is proposed to solve the distribution shift problem. During the policy constraint offline RL training, it is important to ensure the difference between the learned policy and behavior policy within a given threshold. Thus, the learned policy heavily relies on the quality of the behavior policy. However, a problem exists in existing policy constraint methods: if the dataset contains many low-reward transitions, the learned will be contained with a suboptimal reference policy, leading to slow learning speed, low sample efficiency, and inferior performances. This paper shows that the sampling method in policy constraint offline RL that uses all the transitions in the dataset can be improved. A simple but efficient sample filtering method is proposed to improve the sample efficiency and the final performance. First, we evaluate the score of the transitions by average reward and average discounted reward of episodes in the dataset and extract the transition samples of high scores. Second, the high-score transition samples are used to train the offline RL algorithms. We verify the proposed method in a series of offline RL algorithms and benchmark tasks. Experimental results show that the proposed method outperforms baselines.

Figures (7)

• Figure 1: Offline RL.
• Figure 2: Environmental episodes.
• Figure 3: Policy constraint offline deep RL based on sample filtering.
• Figure 4: D4RL environment
• Figure 5: Experimental results of BEAR.