Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios

Songyi Gao, Zuolin Tu, Rong-Jun Qin, Yi-Hao Sun, Xiong-Hui Chen, Yang Yu

TL;DR

NeoRL-2 extends offline reinforcement learning benchmarks with seven near real-world tasks that embed time-delay transitions, external factors, global safety constraints, and data-scarce regimes. The authors use deterministic data collection and PID-based sampling to mimic practical data collection, and evaluate a spectrum of Baseline, Model-Free, and Model-Based offline RL methods. Results show that current SOTA offline RL algorithms often fail to outperform the data-collection policy across tasks, with model-based methods exhibiting notable instability in some environments. Together, NeoRL-2 provides a rigorous, near real-world benchmark to drive development of more robust offline RL methods applicable to real deployments.

Abstract

Offline reinforcement learning (RL) aims to learn from historical data without requiring (costly) access to the environment. To facilitate offline RL research, we previously introduced NeoRL, which highlighted that datasets from real-world tasks are often conservative and limited. With years of experience applying offline RL to various domains, we have identified additional real-world challenges. These include extremely conservative data distributions produced by deployed control systems, delayed action effects caused by high-latency transitions, external factors arising from the uncontrollable variance of transitions, and global safety constraints that are difficult to evaluate during the decision-making process. These challenges are underrepresented in previous benchmarks but frequently occur in real-world tasks. To address this, we constructed the extended Near Real-World Offline RL Benchmark (NeoRL-2), which consists of 7 datasets from 7 simulated tasks along with their corresponding evaluation simulators. Benchmarking results from state-of-the-art offline RL approaches demonstrate that current methods often struggle to outperform the data-collection behavior policy, highlighting the need for more effective methods. We hope NeoRL-2 will accelerate the development of reinforcement learning algorithms for real-world applications. The benchmark project page is available at https://github.com/polixir/NeoRL2.

NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios

TL;DR

NeoRL-2 extends offline reinforcement learning benchmarks with seven near real-world tasks that embed time-delay transitions, external factors, global safety constraints, and data-scarce regimes. The authors use deterministic data collection and PID-based sampling to mimic practical data collection, and evaluate a spectrum of Baseline, Model-Free, and Model-Based offline RL methods. Results show that current SOTA offline RL algorithms often fail to outperform the data-collection policy across tasks, with model-based methods exhibiting notable instability in some environments. Together, NeoRL-2 provides a rigorous, near real-world benchmark to drive development of more robust offline RL methods applicable to real deployments.

Abstract

Offline reinforcement learning (RL) aims to learn from historical data without requiring (costly) access to the environment. To facilitate offline RL research, we previously introduced NeoRL, which highlighted that datasets from real-world tasks are often conservative and limited. With years of experience applying offline RL to various domains, we have identified additional real-world challenges. These include extremely conservative data distributions produced by deployed control systems, delayed action effects caused by high-latency transitions, external factors arising from the uncontrollable variance of transitions, and global safety constraints that are difficult to evaluate during the decision-making process. These challenges are underrepresented in previous benchmarks but frequently occur in real-world tasks. To address this, we constructed the extended Near Real-World Offline RL Benchmark (NeoRL-2), which consists of 7 datasets from 7 simulated tasks along with their corresponding evaluation simulators. Benchmarking results from state-of-the-art offline RL approaches demonstrate that current methods often struggle to outperform the data-collection behavior policy, highlighting the need for more effective methods. We hope NeoRL-2 will accelerate the development of reinforcement learning algorithms for real-world applications. The benchmark project page is available at https://github.com/polixir/NeoRL2.

Paper Structure

This paper contains 23 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Offline RL
  4. Offline RL Algorithm Benchmark
  5. The Reality Gap
  6. Motivation of NeoRL-2
  7. Environment Properties
  8. NeoRL-2 Tasks and Datasets
  9. Tasks
  10. Datasets
  11. Experiments
  12. Comparing Methods
  13. Baseline
  14. Model-Free Methods
  15. Model-Based Methods
  16. ...and 8 more sections

Figures (5)

  • Figure 1: The baseline algorithm normalized scores over seven environments compared to that of offline data.
  • Figure 2: The proportion of baseline algorithms whose scores exceed the data scores.
  • Figure 3: The specific number of tasks in which an algorithm surpasses data scores.
  • Figure 4: The results of all hyperparameter combinations. To enhance the display effect, all scores below -10 are clipped to -10.
  • Figure 5: Comparison of response curves of dynamic model and real environments for DMSD.