A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
Ali Murtaza Caunhye, Asad Jeewa
TL;DR
The paper addresses offline reinforcement learning for continuous control under varying reward densities. It systematically compares Decision Transformer (DT) with traditional offline RL methods, Conservative Q-Learning (CQL) and Implicit Q-Learning (IQL), in the ANT environment across dense and sparse reward settings and different data qualities drawn from D4RL. Key findings show DT exhibits robust, low-variance performance across reward structures and excels in sparse, mixed-quality data (notably medium-expert), while IQL performs best in dense, high-quality data and CQL offers balanced results; DT, however, demands higher computational resources. These results highlight the potential of sequence-modeling approaches for uncertain reward structures and inform practitioners about trade-offs between performance stability and computational cost in offline RL.
Abstract
The field of Offline Reinforcement Learning (RL) aims to derive effective policies from pre-collected datasets without active environment interaction. While traditional offline RL algorithms like Conservative Q-Learning (CQL) and Implicit Q-Learning (IQL) have shown promise, they often face challenges in balancing exploration and exploitation, especially in environments with varying reward densities. The recently proposed Decision Transformer (DT) approach, which reframes offline RL as a sequence modelling problem, has demonstrated impressive results across various benchmarks. This paper presents a comparative study evaluating the performance of DT against traditional offline RL algorithms in dense and sparse reward settings for the ANT continous control environment. Our research investigates how these algorithms perform when faced with different reward structures, examining their ability to learn effective policies and generalize across varying levels of feedback. Through empirical analysis in the ANT environment, we found that DTs showed less sensitivity to varying reward density compared to other methods and particularly excelled with medium-expert datasets in sparse reward scenarios. In contrast, traditional value-based methods like IQL showed improved performance in dense reward settings with high-quality data, while CQL offered balanced performance across different data qualities. Additionally, DTs exhibited lower variance in performance but required significantly more computational resources compared to traditional approaches. These findings suggest that sequence modelling approaches may be more suitable for scenarios with uncertain reward structures or mixed-quality data, while value-based methods remain competitive in settings with dense rewards and high-quality demonstrations.