Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents

TL;DR

This paper addresses sample inefficiency in long-horizon reinforcement learning by leveraging high-level specifications expressed in $SPECTRL$ to decompose tasks into DAG-structured sub-tasks. It introduces Logical Specifications-guided Dynamic Task Sampling ($\mathrm{LSTS}$), a Teacher-Student framework that adaptively samples and learns sub-task policies on the edges of the abstract graph $\mathcal{G}_{\phi}$ without requiring environment dynamics or Reward Machines. Formally, it defines sub-tasks as reachability objectives $Task(q,p)$ and optimizes a sequence of edge policies $\pi^*$ to realize a path from the initial to a final node with probability at least $\eta$, while minimizing total interactions. Across gridworld, robotic, and search-and-rescue tasks, $\mathrm{LSTS}$ and its continuation variant $\mathrm{LSTS}^{ct}$ achieve substantial gains in sample efficiency and faster time-to-threshold compared to automaton-guided RL baselines and curriculum methods, demonstrating practical impact for complex, partially observable, and continuous control domains.

Abstract

Reinforcement Learning (RL) has made significant strides in enabling artificial agents to learn diverse behaviors. However, learning an effective policy often requires a large number of environment interactions. To mitigate sample complexity issues, recent approaches have used high-level task specifications, such as Linear Temporal Logic (LTL$_f$) formulas or Reward Machines (RM), to guide the learning progress of the agent. In this work, we propose a novel approach, called Logical Specifications-guided Dynamic Task Sampling (LSTS), that learns a set of RL policies to guide an agent from an initial state to a goal state based on a high-level task specification, while minimizing the number of environmental interactions. Unlike previous work, LSTS does not assume information about the environment dynamics or the Reward Machine, and dynamically samples promising tasks that lead to successful goal policies. We evaluate LSTS on a gridworld and show that it achieves improved time-to-threshold performance on complex sequential decision-making problems compared to state-of-the-art RM and Automaton-guided RL baselines, such as Q-Learning for Reward Machines and Compositional RL from logical Specifications (DIRL). Moreover, we demonstrate that our method outperforms RM and Automaton-guided RL baselines in terms of sample-efficiency, both in a partially observable robotic task and in a continuous control robotic manipulation task.

Figures (8)

• Figure 1: Gridworld domain
• Figure 2: SPECTRL formula and its corresponding DAG. The DAG excludes all self-loops and transitions to a sink state.
• Figure 3: Learning curve
• Figure 5: Averaged over 10 trials: Learning curves for approaches with converged policies.
• Figure 6: TurtleBot domain