Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning

Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, Gerhard Neumann

TL;DR

TOP-ERL addresses the sample inefficiency of Episodic Reinforcement Learning by introducing a Transformer-based critic that values action sequences. The method partitions long trajectories, trains the Transformer with N-step targets, and updates a Gaussian MP-parameter policy via SAC-style optimization with trust-region enforcement. Empirical results on challenging robotic tasks and the Meta-World MT50 benchmark show improved sample efficiency and robust performance, with ablations highlighting the importance of random segment lengths, initial-condition enforcement, and stable updates. This work broadens the applicability of ERL to off-policy settings and demonstrates the potential of Transformer architectures for sequence-valued policy evaluation in online RL.

Abstract

This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), a novel algorithm that enables off-policy updates in the ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single actions at every time step. These trajectories are typically parameterized by trajectory generators such as Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing high-level temporal correlations. However, ERL methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for entire action sequences, limiting their sample efficiency and preventing the use of more efficient off-policy architectures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-based critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms state-of-the-art RL methods. Thorough ablation studies additionally show the impact of key design choices on the model performance.

TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning

TL;DR

TOP-ERL addresses the sample inefficiency of Episodic Reinforcement Learning by introducing a Transformer-based critic that values action sequences. The method partitions long trajectories, trains the Transformer with N-step targets, and updates a Gaussian MP-parameter policy via SAC-style optimization with trust-region enforcement. Empirical results on challenging robotic tasks and the Meta-World MT50 benchmark show improved sample efficiency and robust performance, with ablations highlighting the importance of random segment lengths, initial-condition enforcement, and stable updates. This work broadens the applicability of ERL to off-policy settings and demonstrates the potential of Transformer architectures for sequence-valued policy evaluation in online RL.

Abstract

This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), a novel algorithm that enables off-policy updates in the ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single actions at every time step. These trajectories are typically parameterized by trajectory generators such as Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing high-level temporal correlations. However, ERL methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for entire action sequences, limiting their sample efficiency and preventing the use of more efficient off-policy architectures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-based critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms state-of-the-art RL methods. Thorough ablation studies additionally show the impact of key design choices on the model performance.

Paper Structure

This paper contains 52 sections, 21 equations, 13 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Preliminaries
  4. Off-Policy Reinforcement Learning
  5. Episodic Reinforcement Learning (ERL)
  6. Transformer-based Off-Policy ERL
  7. Trajectory Generation: Techniques Adopted from ERL Literature
  8. Transformers as value predictor for action sequences
  9. N-step Returns as the target for Transformer Critic
  10. Enforce initial condition for newly predicted action sequence
  11. Policy updates using the Transformer critic
  12. Additional Design Choices from the Literature for Stable Learning
  13. Experiments
  14. Improving Sample Efficiency in Tasks with Challenging Exploration
  15. Consistent Performance in large-scale Manipulation Benchmarks
  16. ...and 37 more sections

Figures (13)

  • Figure 1: Trajectory generation and environment rollout.
  • Figure 2: Architecture overview of the Transformer critic, as described in Sec. \ref{['subsec:transformer_critic']}.
  • Figure 3: Enforce action initial condition
  • Figure 4: Task Evaluation of (a) Metaworld success rate of 50 tasks aggregation. (b) Hopper Jump Max Height. (c) Box Pushing success rate in dense reward, and (d) sparse reward settings.
  • Figure 5: Performance of different critic update strategies (solid lines) and model ablations (dashed lines), using Box pushing dense and sparse reward settings respectively.
  • ...and 8 more figures