Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning

TL;DR

This paper tackles offline reinforcement learning by reframing trajectory modeling to address the stitching problem inherent in return-conditioned sequence models. It introduces Q-value Regularized ConvFormer (QDC), which combines Decision ConvFormer’s local-convolutional trajectory modeling with a dual Q-network trained via a Bellman target to steer actions toward high-value returns. The resulting method achieves strong performance on the D4RL benchmark, notably excelling in Maze2D stitching tasks, and demonstrates robust results across Gym and Adroit domains. The approach reduces reliance on large attention-based transformers while maintaining competitive or superior performance, offering practical benefits for offline RL deployment and trajectory stitching scenarios.

Abstract

As a data-driven paradigm, offline reinforcement learning (Offline RL) has been formulated as sequence modeling, where the Decision Transformer (DT) has demonstrated exceptional capabilities. Unlike previous reinforcement learning methods that fit value functions or compute policy gradients, DT adjusts the autoregressive model based on the expected returns, past states, and actions, using a causally masked Transformer to output the optimal action. However, due to the inconsistency between the sampled returns within a single trajectory and the optimal returns across multiple trajectories, it is challenging to set an expected return to output the optimal action and stitch together suboptimal trajectories. Decision ConvFormer (DC) is easier to understand in the context of modeling RL trajectories within a Markov Decision Process compared to DT. We propose the Q-value Regularized Decision ConvFormer (QDC), which combines the understanding of RL trajectories by DC and incorporates a term that maximizes action values using dynamic programming methods during training. This ensures that the expected returns of the sampled actions are consistent with the optimal returns. QDC achieves excellent performance on the D4RL benchmark, outperforming or approaching the optimal level in all tested environments. It particularly demonstrates outstanding competitiveness in trajectory stitching capability.

Figures (2)

• Figure 1: This is the QDC training framework. It uses state $s$, action $a$, and expected reward $\hat{R}_t$ as inputs to predict the next action $\hat{a}_t$. The predicted action $\hat{a}_t$ and state $s_t$ are then passed to the Q-network to learn the target network. Finally, QDC describes how, based on the Q-network, the policy is guided to take actions consistent with the expected return to stitch different trajectories into the optimal trajectory.
• Figure 2: Ablation on the long task horizon ability. This encompasses the performance comparison of different input sequence horizons $K \in [10, 80]$ in the walker2d-medium-replay-v2 task.