Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Reinforcement Learning in Probabilistic Reward Machines

Xiaofeng Lin, Xuezhou Zhang

TL;DR

An algorithm is designed for PRMs that achieves a regret bound of Õ((HOAT)^(1/2) + H²O²A^(3/2) + H(T)^(1/2)), where H is the time horizon, O is the number of observations, A is the number of actions, and T is the number of time steps.

Abstract

In this paper, we study reinforcement learning in Markov Decision Processes with Probabilistic Reward Machines (PRMs), a form of non-Markovian reward commonly found in robotics tasks. We design an algorithm for PRMs that achieves a regret bound of $\widetilde{O}(\sqrt{HOAT} + H^2O^2A^{3/2} + H\sqrt{T})$, where $H$ is the time horizon, $O$ is the number of observations, $A$ is the number of actions, and $T$ is the number of time-steps. This result improves over the best-known bound, $\widetilde{O}(H\sqrt{OAT})$ of \citet{pmlr-v206-bourel23a} for MDPs with Deterministic Reward Machines (DRMs), a special case of PRMs. When $T \geq H^3O^3A^2$ and $OA \geq H$, our regret bound leads to a regret of $\widetilde{O}(\sqrt{HOAT})$, which matches the established lower bound of $Ω(\sqrt{HOAT})$ for MDPs with DRMs up to a logarithmic factor. To the best of our knowledge, this is the first efficient algorithm for PRMs. Additionally, we present a new simulation lemma for non-Markovian rewards, which enables reward-free exploration for any non-Markovian reward given access to an approximate planner. Complementing our theoretical findings, we show through extensive experiment evaluations that our algorithm indeed outperforms prior methods in various PRM environments.

Efficient Reinforcement Learning in Probabilistic Reward Machines

TL;DR

An algorithm is designed for PRMs that achieves a regret bound of Õ((HOAT)^(1/2) + H²O²A^(3/2) + H(T)^(1/2)), where H is the time horizon, O is the number of observations, A is the number of actions, and T is the number of time steps.

Abstract

In this paper, we study reinforcement learning in Markov Decision Processes with Probabilistic Reward Machines (PRMs), a form of non-Markovian reward commonly found in robotics tasks. We design an algorithm for PRMs that achieves a regret bound of , where is the time horizon, is the number of observations, is the number of actions, and is the number of time-steps. This result improves over the best-known bound, of \citet{pmlr-v206-bourel23a} for MDPs with Deterministic Reward Machines (DRMs), a special case of PRMs. When and , our regret bound leads to a regret of , which matches the established lower bound of for MDPs with DRMs up to a logarithmic factor. To the best of our knowledge, this is the first efficient algorithm for PRMs. Additionally, we present a new simulation lemma for non-Markovian rewards, which enables reward-free exploration for any non-Markovian reward given access to an approximate planner. Complementing our theoretical findings, we show through extensive experiment evaluations that our algorithm indeed outperforms prior methods in various PRM environments.
Paper Structure (33 sections, 28 theorems, 132 equations, 4 figures, 1 table, 4 algorithms)

This paper contains 33 sections, 28 theorems, 132 equations, 4 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reward Machines
  4. Reward-free exploration
  5. Problem Formulation
  6. Labeled MDPs with Probabilistic Reward Machines
  7. Learning Algorithms and Results
  8. RL with Non-Markoivian Rewards
  9. Exploration stage
  10. Planning stage
  11. Experiments
  12. DRM Experiments
  13. PRM Experiments
  14. Conclusion
  15. Table of notation
  16. ...and 18 more sections

Key Result

Lemma 1

pmlr-v206-bourel23a Let $M = ({\cal O}, {\cal A}, p, {\cal R},{\cal P}, L, H)$ be a finite MDP with PRM. Then, an associated cross-product MDP to $M$ is $M_{cp} = ({\cal S}, \mathcal{A}, P, R)$, where ${\cal S} = {\cal Q} \times \mathcal{O}$ and for $s = (q, o)$, $s' = (q', o') \in {\cal S}$ and $a

Figures (4)

  • Figure 1: The Warehouse example and the corresponding PRM. The robot needs to pick up an item and delivers the item to the right location in sequence when the item may not be ready and the delivery location could be busy. (a): a $4\times4$ grid world in which is our robot, is the charging station, is the item pickup location, is the delivery location;(b): The corresponding PRM, where an edge $q \xrightarrow[r]{\ell \mid p} q'$ represents that state $q$ transitions to $q'$ on event $l$ with probability $p$ and receives reward $r$.
  • Figure 2: The labeled RiverSwim and the corresponding DRM.
  • Figure 3: Experimental results in RiverSwim
  • Figure 4: Experimental results in Warehouse

Theorems & Definitions (50)

  • Lemma 1
  • Theorem 1
  • Definition 1
  • Theorem 2
  • Lemma 2
  • Definition 2: Significant Observation
  • Theorem 3
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • ...and 40 more