Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control

Nikita Kachaev, Daniil Zelezetsky, Egor Cherepanov, Alexey K. Kovelev, Aleksandr I. Panov

TL;DR

The paper addresses the under-exploration of transformers in online model-free reinforcement learning for continuous control and investigates practical design choices to make them effective. It presents a unified training pipeline and systematically studies transformer conditioning, actor–critic backbone sharing, and data slicing across MDPs, POMDPs, vector-based, and image-based tasks, demonstrating competitive performance with carefully tuned setups. The key contributions include actionable guidelines such as EmbedConcat conditioning, maintaining separate backbones with critic-freezing, and cross-episode data slicing, validated on MuJoCo and ManiSkill3 benchmarks. This work broadens the applicability of transformers in online RL and provides a concrete pathway for practitioners to leverage their long-horizon modeling capabilities in real-time control and sim-to-real transfer.

Abstract

Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL.

A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control

TL;DR

The paper addresses the under-exploration of transformers in online model-free reinforcement learning for continuous control and investigates practical design choices to make them effective. It presents a unified training pipeline and systematically studies transformer conditioning, actor–critic backbone sharing, and data slicing across MDPs, POMDPs, vector-based, and image-based tasks, demonstrating competitive performance with carefully tuned setups. The key contributions include actionable guidelines such as EmbedConcat conditioning, maintaining separate backbones with critic-freezing, and cross-episode data slicing, validated on MuJoCo and ManiSkill3 benchmarks. This work broadens the applicability of transformers in online RL and provides a concrete pathway for practitioners to leverage their long-horizon modeling capabilities in real-time control and sim-to-real transfer.

Abstract

Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL.
Paper Structure (24 sections, 17 figures, 7 tables)

This paper contains 24 sections, 17 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Transformers in RL
  3. Offline RL
  4. Hybrid Methods
  5. Online RL
  6. Methodology
  7. Environments
  8. Transformer Baselines
  9. Experimental protocol
  10. Experiments and Results
  11. RQ1: How does transformer conditioning affect performance?
  12. Interpretation
  13. RQ2: How does actor–critic backbone sharing affect training stability and efficiency?
  14. RQ3: How does input data slicing affect the training of transformers?
  15. Comparison with Baselines
  16. ...and 9 more sections

Figures (17)

  • Figure 1: Transformer baselines enhanced by our findings achieve competitive performance across selected tasks. Average reward is normalized over the specified number of tasks. For MuJoCo-POMDP, we use PPO-GRU from luo2024efficientrecurrentoffpolicyrl.
  • Figure 2: Scheme of the GPT-2-like block used as a backbone during all the experiments in this work.
  • Figure 3: Performance of transformer conditioning methods on MuJoCo-POMDP tasks, where velocity is masked to induce partial observability. The two CrossAttn variants differ in sequence input order.
  • Figure 4: Conditioning experiments for MDP version of MuJoCo tasks. Using observations only is enough for tasks with Markovian properties.
  • Figure 5: t-SNE visualization of action representations from the transformer on HalfCheetah. Clusters with EmbedConcat appear more structured.
  • ...and 12 more figures