Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks

Ruixuan Miao, Xu Lu, Cong Tian, Bin Yu, Zhenhua Duan

TL;DR

ParMod tackles learning non-Markovian tasks by translating temporal logic specifications into DFAs and constructing a product MDP to encode memory into state space. It then partitions the DFA into task phases via Task Phase Classification and trains each phase in parallel with dedicated networks, using a DFA-informed reward shaping scheme to improve sample efficiency. The framework is theoretically grounded in NM–to–MDP equivalence and optimality results, and empirically demonstrates faster learning, higher success rates, and better policy quality than baselines across Waterworld, Racecar, and Halfcheetah benchmarks. The approach offers scalable, parallelizable handling of temporally extended tasks and shows promise for distributed implementations and dynamic phase classification in future work.

Abstract

The commonly used Reinforcement Learning (RL) model, MDPs (Markov Decision Processes), has a basic premise that rewards depend on the current state and action only. However, many real-world tasks are non-Markovian, which has long-term memory and dependency. The reward sparseness problem is further amplified in non-Markovian scenarios. Hence learning a non-Markovian task (NMT) is inherently more difficult than learning a Markovian one. In this paper, we propose a novel \textbf{Par}allel and \textbf{Mod}ular RL framework, ParMod, specifically for learning NMTs specified by temporal logic. With the aid of formal techniques, the NMT is modulaized into a series of sub-tasks based on the automaton structure (equivalent to its temporal logic counterpart). On this basis, sub-tasks will be trained by a group of agents in a parallel fashion, with one agent handling one sub-task. Besides parallel training, the core of ParMod lies in: a flexible classification method for modularizing the NMT, and an effective reward shaping method for improving the sample efficiency. A comprehensive evaluation is conducted on several challenging benchmark problems with respect to various metrics. The experimental results show that ParMod achieves superior performance over other relevant studies. Our work thus provides a good synergy among RL, NMT and temporal logic.

ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks

TL;DR

ParMod tackles learning non-Markovian tasks by translating temporal logic specifications into DFAs and constructing a product MDP to encode memory into state space. It then partitions the DFA into task phases via Task Phase Classification and trains each phase in parallel with dedicated networks, using a DFA-informed reward shaping scheme to improve sample efficiency. The framework is theoretically grounded in NM–to–MDP equivalence and optimality results, and empirically demonstrates faster learning, higher success rates, and better policy quality than baselines across Waterworld, Racecar, and Halfcheetah benchmarks. The approach offers scalable, parallelizable handling of temporally extended tasks and shows promise for distributed implementations and dynamic phase classification in future work.

Abstract

The commonly used Reinforcement Learning (RL) model, MDPs (Markov Decision Processes), has a basic premise that rewards depend on the current state and action only. However, many real-world tasks are non-Markovian, which has long-term memory and dependency. The reward sparseness problem is further amplified in non-Markovian scenarios. Hence learning a non-Markovian task (NMT) is inherently more difficult than learning a Markovian one. In this paper, we propose a novel \textbf{Par}allel and \textbf{Mod}ular RL framework, ParMod, specifically for learning NMTs specified by temporal logic. With the aid of formal techniques, the NMT is modulaized into a series of sub-tasks based on the automaton structure (equivalent to its temporal logic counterpart). On this basis, sub-tasks will be trained by a group of agents in a parallel fashion, with one agent handling one sub-task. Besides parallel training, the core of ParMod lies in: a flexible classification method for modularizing the NMT, and an effective reward shaping method for improving the sample efficiency. A comprehensive evaluation is conducted on several challenging benchmark problems with respect to various metrics. The experimental results show that ParMod achieves superior performance over other relevant studies. Our work thus provides a good synergy among RL, NMT and temporal logic.

Paper Structure

This paper contains 15 sections, 5 theorems, 16 equations, 13 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Solutions for Non-Markovian Tasks
  4. Distributed Reinforcement Learning
  5. Preliminaries
  6. Linear Temporal Logic over Finite Traces
  7. MDP and NMRDP
  8. Parallel and Modular RL Framework
  9. Modularizing Non-Markovian Tasks
  10. Parallel Training for Modular Task Phases
  11. Correctness and Optimality
  12. Evaluation
  13. Experiment Setup
  14. Experimental Results
  15. Summary and Discussion

Key Result

Theorem 1

An NMRDP $\text{NM}= \langle S, A, R=\langle \varphi, r \rangle, P, \gamma, s_{0} \rangle$ is equivalent to the product MDP $M^\otimes =\left \langle S^\otimes,A,R^\otimes,P^\otimes,\gamma,s^\otimes_0\right \rangle$, which is the product of $\text{NM}$ and the DFA $\mathcal{A} =\langle Q,\Sigma,\del

Figures (13)

  • Figure 1: An instance of the Waterworld problem
  • Figure 2: Overview of the framework ParMod
  • Figure 3: DFA of $\varphi = \left ( \neg r \wedge \neg g \right ) \text{U}((r \wedge \neg g) \wedge \bigcirc ((\neg r \wedge \neg g)\text{U} (g \wedge \neg r)))$. $\neg g \wedge \neg r, g \wedge \neg r, r \wedge \neg g$ are logical representations of $\{g, r\}, \{ g\}, \{ r \}$ respectively. $g$ represents $\{ g, r \}$ or $\{ g \}$, similar to $r$.
  • Figure 4: Relationship of task phases in ParMod
  • Figure 5: Depiction of the Racecar problem
  • ...and 8 more figures

Theorems & Definitions (8)

  • Example 1: Waterworld problem - detailed
  • Example 2: rank of DFA states
  • Definition 1
  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Theorem 3
  • Theorem 4