Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning

Zijian Guo, Weichao Zhou, Wenchao Li

TL;DR

This work tackles offline safe reinforcement learning with complex temporal constraints by introducing the Specification-conditioned Decision Transformer (SDT), which conditions trajectory generation on Signal Temporal Logic (STL) specifications. SDT extends the Decision Transformer with two robustness tokens, prefix and suffix, based on STL robustness $\rho(\tau,t,\phi)$, enabling non-Markovian constraint handling while preserving high reward. Empirical results on DSRL benchmarks show that SDT outperforms constrained-optimization and conditioned-RL baselines in both safety (satisfaction rate and cost) and reward, and can adapt to different target suffix configurations without retraining. The approach demonstrates strong alignment between actual and target suffix values and confirms the critical role of both prefix and suffix robustness in learning safe, high-performing policies, with robustness to predicate rescaling observed in experiments.

Abstract

Offline safe reinforcement learning (RL) aims to train a constraint satisfaction policy from a fixed dataset. Current state-of-the-art approaches are based on supervised learning with a conditioned policy. However, these approaches fall short in real-world applications that involve complex tasks with rich temporal and logical structures. In this paper, we propose temporal logic Specification-conditioned Decision Transformer (SDT), a novel framework that harnesses the expressive power of signal temporal logic (STL) to specify complex temporal rules that an agent should follow and the sequential modeling capability of Decision Transformer (DT). Empirical evaluations on the DSRL benchmarks demonstrate the better capacity of SDT in learning safe and high-reward policies compared with existing approaches. In addition, SDT shows good alignment with respect to different desired degrees of satisfaction of the STL specification that it is conditioned on.

Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning

TL;DR

This work tackles offline safe reinforcement learning with complex temporal constraints by introducing the Specification-conditioned Decision Transformer (SDT), which conditions trajectory generation on Signal Temporal Logic (STL) specifications. SDT extends the Decision Transformer with two robustness tokens, prefix and suffix, based on STL robustness , enabling non-Markovian constraint handling while preserving high reward. Empirical results on DSRL benchmarks show that SDT outperforms constrained-optimization and conditioned-RL baselines in both safety (satisfaction rate and cost) and reward, and can adapt to different target suffix configurations without retraining. The approach demonstrates strong alignment between actual and target suffix values and confirms the critical role of both prefix and suffix robustness in learning safe, high-performing policies, with robustness to predicate rescaling observed in experiments.

Abstract

Offline safe reinforcement learning (RL) aims to train a constraint satisfaction policy from a fixed dataset. Current state-of-the-art approaches are based on supervised learning with a conditioned policy. However, these approaches fall short in real-world applications that involve complex tasks with rich temporal and logical structures. In this paper, we propose temporal logic Specification-conditioned Decision Transformer (SDT), a novel framework that harnesses the expressive power of signal temporal logic (STL) to specify complex temporal rules that an agent should follow and the sequential modeling capability of Decision Transformer (DT). Empirical evaluations on the DSRL benchmarks demonstrate the better capacity of SDT in learning safe and high-reward policies compared with existing approaches. In addition, SDT shows good alignment with respect to different desired degrees of satisfaction of the STL specification that it is conditioned on.
Paper Structure (18 sections, 11 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 18 sections, 11 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Offline Safe RL
  5. Decision Transformers
  6. STL Specification
  7. Method
  8. Offline RL via Supervised Learning
  9. Specification-Conditioned Decision Transformers
  10. Suffix and Prefix Robustness Values
  11. Experiment
  12. Can SDT learn specification-satisfying policies?
  13. Can SDT adapt to different target suffixes?
  14. Ablation studies
  15. Conclusion
  16. ...and 3 more sections

Figures (7)

  • Figure 1: The SDT framework. It takes the prefix and suffix robustness values, return-to-go, states, and actions as inputs and predicts the next actions using a Gaussian policy.
  • Figure 2: Results of alignment with different target suffixes. The top-row plots show the evaluated reward and the bottom-row plots show the evaluated suffix. The solid line and the light shade area represent the mean and mean $\pm$ standard deviation.
  • Figure 3: Evaluation results of normalized rewards in D4RL Gym environments. Each value is averaged over 3 seeds. m: medium, mr: medium-replay, me: medium-expert.
  • Figure 4: Ablation study. Left: the effect of the prefix and the suffix. Right: influence of different target suffix configurations.
  • Figure 5: Illustration of the offline dataset. The first-row plots show the relabeled cost versus reward and the second-row plots show the suffix ($\rho_{suf}(\tau_{1:T}, 1, \phi)$) versus reward. Each column represents an environment. Each point denotes a collected trajectory (not necessarily to be unique) with corresponding episodic relabeled cost (or suffix) and reward value.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Definition 4.1