Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Efficient Exploration in Reward Machines with Low Regret

Hippolyte Bourel, Anders Jonsson, Odalric-Ambrym Maillard, Chenxiao Ma, Mohammad Sadegh Talebi

TL;DR

This paper studies reinforcement learning for average-reward tasks where rewards depend on non-Markovian histories via reward machines. It formalizes the MDPRM framework and leverages a cross-product MDP $M^×$ to design optimistic, model-based learners that exploit RM structure. The key contributions are two UCRL-style algorithms, UCRL-PRM-L1 and UCRL-PRM-B, with regret guarantees that adapt to RM sparsity through the RM-restricted diameter, and a regret lower bound for deterministic RM settings. These results improve over structure-agnostic baselines and provide a principled approach to exploration in history-dependent tasks modeled by reward machines.

Abstract

We study reinforcement learning (RL) for decision processes with non-Markovian reward, in which high-level knowledge of the task in the form of reward machines is available to the learner. We consider probabilistic reward machines with initially unknown dynamics, and investigate RL under the average-reward criterion, where the learning performance is assessed through the notion of regret. Our main algorithmic contribution is a model-based RL algorithm for decision processes involving probabilistic reward machines that is capable of exploiting the structure induced by such machines. We further derive high-probability and non-asymptotic bounds on its regret and demonstrate the gain in terms of regret over existing algorithms that could be applied, but obliviously to the structure. We also present a regret lower bound for the studied setting. To the best of our knowledge, the proposed algorithm constitutes the first attempt to tailor and analyze regret specifically for RL with probabilistic reward machines.

Provably Efficient Exploration in Reward Machines with Low Regret

TL;DR

This paper studies reinforcement learning for average-reward tasks where rewards depend on non-Markovian histories via reward machines. It formalizes the MDPRM framework and leverages a cross-product MDP to design optimistic, model-based learners that exploit RM structure. The key contributions are two UCRL-style algorithms, UCRL-PRM-L1 and UCRL-PRM-B, with regret guarantees that adapt to RM sparsity through the RM-restricted diameter, and a regret lower bound for deterministic RM settings. These results improve over structure-agnostic baselines and provide a principled approach to exploration in history-dependent tasks modeled by reward machines.

Abstract

We study reinforcement learning (RL) for decision processes with non-Markovian reward, in which high-level knowledge of the task in the form of reward machines is available to the learner. We consider probabilistic reward machines with initially unknown dynamics, and investigate RL under the average-reward criterion, where the learning performance is assessed through the notion of regret. Our main algorithmic contribution is a model-based RL algorithm for decision processes involving probabilistic reward machines that is capable of exploiting the structure induced by such machines. We further derive high-probability and non-asymptotic bounds on its regret and demonstrate the gain in terms of regret over existing algorithms that could be applied, but obliviously to the structure. We also present a regret lower bound for the studied setting. To the best of our knowledge, the proposed algorithm constitutes the first attempt to tailor and analyze regret specifically for RL with probabilistic reward machines.
Paper Structure (49 sections, 19 theorems, 104 equations, 5 figures, 4 algorithms)

This paper contains 49 sections, 19 theorems, 104 equations, 5 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Organization
  5. Problem Formulation
  6. MDPRMs: Average-Reward Markov Decision Processes with Reward Machines
  7. Labeled Markov Decision Processes
  8. (Probabilistic) Reward Machines
  9. Average-Reward MDPs with RMs
  10. Regret Minimization in MDPRMs
  11. Learning Algorithms for MDPRMs
  12. Confidence Sets
  13. Confidence Sets for Observation Dynamics $P$
  14. Confidence Sets for RM State Dynamics $\tau$
  15. Set of Plausible Models
  16. ...and 34 more sections

Key Result

Lemma 1

Let $M\!=\!(\mathcal{O},\mathcal{A},P,\mathcal{R},\emph{AP},L)$ be a finite MDPRM. Then, an associated cross-product MDP to $M$ is $M^\times\!=\!(\mathcal{S},\mathcal{A},P^\times,R^\times)$, with $\mathcal{S}=\mathcal{Q}\times\mathcal{O}$, where for $s\!=\!(q,o), s'\!=\!(q',o')\!\in\!\mathcal{S}$, a

Figures (5)

  • Figure 1: Interaction with an MDPRM
  • Figure 2: An example environment with one RM.
  • Figure 3: An example where RM-restricted diameter $D_s \lesssim D^\times/Q$. The labeled MDP in left, and the RM in right.
  • Figure 4: Construction of the underlying labeled MDP for the LB with $A = 2$ and $O = 8$, based on the worst-case MDP in lattimore2020bandit.
  • Figure 5: Construction of the underlying RM for the lower bound with a double-cyclic a 'good' cycle giving rewards and 'bad' cycle of similar length giving no reward.

Theorems & Definitions (22)

  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Definition 1: jaksch2010near
  • Definition 2: RM-Restricted Diameter
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • ...and 12 more