TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks

TL;DR

This work tackles the challenge of learning control policies for long-horizon tasks specified in Signal Temporal Logic (STL), where STL's history dependence and sparse rewards hinder standard RL. It introduces TGPO, a hierarchical framework that ground STL into timed subgoals (Reach and Invariance) and uses a high-level time allocation alongside a low-level time-conditioned policy trained with dense, stage-wise rewards. A critic-guided Bayesian time allocation via Metropolis-Hastings sampling focuses exploration on temporally feasible schedules, while an augmented MDP and PPO-based learning drive the policy. Experiments across five robotic environments demonstrate that TGPO, and particularly TGPO*, achieve superior task success rates, especially for high-dimensional and long-horizon STL tasks, with interpretable critic-driven planning and diverse multi-modal behaviors. The paper contributes a general STL-solving framework, a critic-guided temporal grounding mechanism, and interpretable learning dynamics, with open-source code to enable reproducibility and further research.

Abstract

Learning control policies for complex, long-horizon tasks is a central challenge in robotics and autonomous systems. Signal Temporal Logic (STL) offers a powerful and expressive language for specifying such tasks, but its non-Markovian nature and inherent sparse reward make it difficult to be solved via standard Reinforcement Learning (RL) algorithms. Prior RL approaches focus only on limited STL fragments or use STL robustness scores as sparse terminal rewards. In this paper, we propose TGPO, Temporal Grounded Policy Optimization, to solve general STL tasks. TGPO decomposes STL into timed subgoals and invariant constraints and provides a hierarchical framework to tackle the problem. The high-level component of TGPO proposes concrete time allocations for these subgoals, and the low-level time-conditioned policy learns to achieve the sequenced subgoals using a dense, stage-wise reward signal. During inference, we sample various time allocations and select the most promising assignment for the policy network to rollout the solution trajectory. To foster efficient policy learning for complex STL with multiple subgoals, we leverage the learned critic to guide the high-level temporal search via Metropolis-Hastings sampling, focusing exploration on temporally feasible solutions. We conduct experiments on five environments, ranging from low-dimensional navigation to manipulation, drone, and quadrupedal locomotion. Under a wide range of STL tasks, TGPO significantly outperforms state-of-the-art baselines (especially for high-dimensional and long-horizon cases), with an average of 31.6% improvement in task success rate compared to the best baseline. The code will be available at https://github.com/mengyuest/TGPO

Figures (36)

• Figure 1: Framework: STL decomposition and critic-guided temporal grounding yield subgoals and invariant constraints that guide an augmented MDP with dense rewards for policy optimization.
• Figure 2: STL decomposition of $\phi=F_{[a,b]}(\mu_1 \land G_{[a_2,b_2]}\mu_2\land F_{[a_3,b_3]}\mu_3) \land G_{[c,d]}\neg \mu_0$.
• Figure 3: Simulation benchmarks.
• Figure 4: Main comparison. Our method has higher task success rate compared to other baselines.
• Figure 5: Main comparison for the STL success rate evaluation along the training process.