Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Expressive Models Truly Necessary for Offline RL?

Guan Wang, Haoyi Niu, Jianxiong Li, Li Jiang, Jianming Hu, Xianyuan Zhan

TL;DR

This paper tackles offline RL by questioning whether highly expressive models are necessary for long-horizon planning. It introduces Recursive Skip-Step Planning (RSP), a lightweight framework that uses recursive coarse-grained dynamics with 2-layer MLPs and a supervised, goal-conditioned policy to achieve strong long-horizon performance. RSP delivers competitive or superior results on the D4RL benchmark, with dramatically reduced training time and inference latency compared to Transformer- or diffusion-based methods. By demonstrating robust performance with simple models, the work advocates rethinking offline RL design toward efficient hierarchical planning rather than ever heavier architectures.

Abstract

Among various branches of offline reinforcement learning (RL) methods, goal-conditioned supervised learning (GCSL) has gained increasing popularity as it formulates the offline RL problem as a sequential modeling task, therefore bypassing the notoriously difficult credit assignment challenge of value learning in conventional RL paradigm. Sequential modeling, however, requires capturing accurate dynamics across long horizons in trajectory data to ensure reasonable policy performance. To meet this requirement, leveraging large, expressive models has become a popular choice in recent literature, which, however, comes at the cost of significantly increased computation and inference latency. Contradictory yet promising, we reveal that lightweight models as simple as shallow 2-layer MLPs, can also enjoy accurate dynamics consistency and significantly reduced sequential modeling errors against large expressive models by adopting a simple recursive planning scheme: recursively planning coarse-grained future sub-goals based on current and target information, and then executes the action with a goal-conditioned policy learned from data rela-beled with these sub-goal ground truths. We term our method Recursive Skip-Step Planning (RSP). Simple yet effective, RSP enjoys great efficiency improvements thanks to its lightweight structure, and substantially outperforms existing methods, reaching new SOTA performances on the D4RL benchmark, especially in multi-stage long-horizon tasks.

Are Expressive Models Truly Necessary for Offline RL?

TL;DR

This paper tackles offline RL by questioning whether highly expressive models are necessary for long-horizon planning. It introduces Recursive Skip-Step Planning (RSP), a lightweight framework that uses recursive coarse-grained dynamics with 2-layer MLPs and a supervised, goal-conditioned policy to achieve strong long-horizon performance. RSP delivers competitive or superior results on the D4RL benchmark, with dramatically reduced training time and inference latency compared to Transformer- or diffusion-based methods. By demonstrating robust performance with simple models, the work advocates rethinking offline RL design toward efficient hierarchical planning rather than ever heavier architectures.

Abstract

Among various branches of offline reinforcement learning (RL) methods, goal-conditioned supervised learning (GCSL) has gained increasing popularity as it formulates the offline RL problem as a sequential modeling task, therefore bypassing the notoriously difficult credit assignment challenge of value learning in conventional RL paradigm. Sequential modeling, however, requires capturing accurate dynamics across long horizons in trajectory data to ensure reasonable policy performance. To meet this requirement, leveraging large, expressive models has become a popular choice in recent literature, which, however, comes at the cost of significantly increased computation and inference latency. Contradictory yet promising, we reveal that lightweight models as simple as shallow 2-layer MLPs, can also enjoy accurate dynamics consistency and significantly reduced sequential modeling errors against large expressive models by adopting a simple recursive planning scheme: recursively planning coarse-grained future sub-goals based on current and target information, and then executes the action with a goal-conditioned policy learned from data rela-beled with these sub-goal ground truths. We term our method Recursive Skip-Step Planning (RSP). Simple yet effective, RSP enjoys great efficiency improvements thanks to its lightweight structure, and substantially outperforms existing methods, reaching new SOTA performances on the D4RL benchmark, especially in multi-stage long-horizon tasks.

Paper Structure

This paper contains 21 sections, 8 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. RL via Goal-Conditioned Supervised Learning
  5. Recursive Skip-Step Planning
  6. Why Non-recursive Planning Might Struggle?
  7. Coarse-grained Planning in a Recursive Way
  8. Policy Extraction
  9. Discussions
  10. Experiments
  11. D4RL Benchmark Results
  12. Evaluation setups
  13. Baselines
  14. Main results
  15. Training Cost and Inference Latency
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Existing GCSL methods trade off score performances for slow training and inference. However, RSP with recursion depth $N=1$ significantly outperforms all baselines on Antmaze datasets while completing training in just 180 seconds and enjoying an inference latency of 0.33ms. $N=4$ further enhances performance with minimal cost, as the inference time increases to 0.48 ms and the training time to 271s, remaining orders of magnitude lower than other methods that use expressive models.
  • Figure 2: The illustration of Recursive Skip-Step Planning
  • Figure 3: Root mean squared error (RMSE) between rollout sub-goal predictions and ground truths on Antmaze-Ultra.
  • Figure 4: Ablations on choices of recursion depth and horizon.