Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications

Jun Wang, Hosein Hasanbeig, Kaiyuan Tan, Zihe Sun, Yiannis Kantaros

TL;DR

A novel Deep Q-learning algorithm is introduced that significantly improves learning speed and generates enhanced sample efficiency from a mission-driven exploration strategy that prioritizes exploration towards directions likely to contribute to mission success.

Abstract

This paper addresses the problem of designing control policies for agents with unknown stochastic dynamics and control objectives specified using Linear Temporal Logic (LTL). Recent Deep Reinforcement Learning (DRL) algorithms have aimed to compute policies that maximize the satisfaction probability of LTL formulas, but they often suffer from slow learning performance. To address this, we introduce a novel Deep Q-learning algorithm that significantly improves learning speed. The enhanced sample efficiency stems from a mission-driven exploration strategy that prioritizes exploration towards directions likely to contribute to mission success. Identifying these directions relies on an automaton representation of the LTL task as well as a learned neural network that partially models the agent-environment interaction. We provide comparative experiments demonstrating the efficiency of our algorithm on robot navigation tasks in unseen environments.

Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications

TL;DR

A novel Deep Q-learning algorithm is introduced that significantly improves learning speed and generates enhanced sample efficiency from a mission-driven exploration strategy that prioritizes exploration towards directions likely to contribute to mission success.

Abstract

This paper addresses the problem of designing control policies for agents with unknown stochastic dynamics and control objectives specified using Linear Temporal Logic (LTL). Recent Deep Reinforcement Learning (DRL) algorithms have aimed to compute policies that maximize the satisfaction probability of LTL formulas, but they often suffer from slow learning performance. To address this, we introduce a novel Deep Q-learning algorithm that significantly improves learning speed. The enhanced sample efficiency stems from a mission-driven exploration strategy that prioritizes exploration towards directions likely to contribute to mission success. Identifying these directions relies on an automaton representation of the LTL task as well as a learned neural network that partially models the agent-environment interaction. We provide comparative experiments demonstrating the efficiency of our algorithm on robot navigation tasks in unseen environments.
Paper Structure (11 sections, 2 figures, 2 algorithms)

This paper contains 11 sections, 2 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Accelerated Deep Reinforcement Learning with Temporal Logic Specifications
  4. Converting LTL formulas into Automata
  5. Product MDP
  6. Accelerated Policy Learning for LTL-encoded Tasks
  7. Computation of the Biased Action
  8. Experiments
  9. Setting Up Experiments
  10. Comparative Numerical Experiments
  11. Conclusions & Future Work

Figures (2)

  • Figure 1: Graphical illustration of the environments. Fig. 1(a) and 1(b) show example environments for Case Studies I, III and IV; Fig 1(c) and 1(d) show example environments for Case Study II.
  • Figure 2: Illustration of the evaluation metrics in each case study. Each row represents a single case study. Columns 1–3 plot metrics (i), (ii), and (iii), respectively. Line colors: Ours (blue), DQN (orange), PPO (green), and SAC (red).

Theorems & Definitions (4)

  • Definition 1: MDP
  • Definition 3: DRA baier2008principles
  • Definition 4: PMDP
  • Remark 5: Training Dataset to Bootstrap RL