Reinforcement learning with timed constraints for robotics motion planning

TL;DR

This work tackles the problem of learning policies for robotic motion planning under explicit time-bounded requirements. It introduces a unified automata-based RL framework that translates MITL into a Timed-LDGBA and synchronizes it with either an MDP or a POMDP to form a product timed model suitable for Q-learning. The approach demonstrates consistent satisfaction of time constraints across stochastic dynamics and partial observability, validated through three simulation studies that scale to larger state spaces and more complex environments. The methodology enables reliable, temporally guaranteed planning in uncertain settings and opens avenues for extensions to multi-agent and hierarchical planning with timed objectives, supported by a straightforward reward structure that enforces MITL satisfaction.

Abstract

Robotic systems operating in dynamic and uncertain environments increasingly require planners that satisfy complex task sequences while adhering to strict temporal constraints. Metric Interval Temporal Logic (MITL) offers a formal and expressive framework for specifying such time-bounded requirements; however, integrating MITL with reinforcement learning (RL) remains challenging due to stochastic dynamics and partial observability. This paper presents a unified automata-based RL framework for synthesizing policies in both Markov Decision Processes (MDPs) and Partially Observable Markov Decision Processes (POMDPs) under MITL specifications. MITL formulas are translated into Timed Limit-Deterministic Generalized Büchi Automata (Timed-LDGBA) and synchronized with the underlying decision process to construct product timed models suitable for Q-learning. A simple yet expressive reward structure enforces temporal correctness while allowing additional performance objectives. The approach is validated in three simulation studies: a $5 \times 5$ grid-world formulated as an MDP, a $10 \times 10$ grid-world formulated as a POMDP, and an office-like service-robot scenario. Results demonstrate that the proposed framework consistently learns policies that satisfy strict time-bounded requirements under stochastic transitions, scales to larger state spaces, and remains effective in partially observable environments, highlighting its potential for reliable robotic planning in time-critical and uncertain settings.

Figures (9)

• Figure 1: An example of a three-state T-LDGBA.
• Figure 2: A 5 $\times$ 5 grid world, with two labels (a) in blue and (b) in green and one block in black.
• Figure 3: The average cumulative reward in the 5 $\times$ 5 grid world without a movement penalty
• Figure 4: A 5 $\times$ 5 grid-world environment illustrating agent trajectories: a) visiting label '$a$', and b) visiting label '$b$'. The Transition is deterministic, and the reward of visiting labels is 100 with no movement penalty.
• Figure 5: A 5 $\times$ 5 grid-world environment illustrating agent trajectories: a) visiting label '$a$', and b) visiting label '$b$'. The Transition is deterministic, and the reward of visiting labels is 100 with the movement penalty.

Theorems & Definitions (6)

• Definition 2.1: MDP
• Definition 2.2: POMDP
• Definition 3.1: MITL
• Definition 3.2: T-LDGBA
• Definition 3.3: Product timed MDP
• Definition 3.4: Product timed POMDP