Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

About Time: Model-free Reinforcement Learning with Timed Reward Machines

Anirban Majumdar, Ritam Raha, Rajarshi Roy, David Parker, Marta Kwiatkowska

TL;DR

The paper introduces Timed Reward Machines (TRMs) to express timing-sensitive, history-dependent rewards in reinforcement learning. It develops model-free Q-learning algorithms by constructing cross-product MDPs that combine the environment with TRMs under digital and real-time semantics, leveraging region/corner abstractions and counterfactual-imagining to manage timing constraints. Empirical results on Taxi and Frozen Lake show that TRMs improve learning when timing matters and that corner-point abstractions often outperform naive discretizations. The work lays groundwork for time-aware RL and points to future extensions to continuous-time models and priced zones for richer timing specifications.

Abstract

Reward specification plays a central role in reinforcement learning (RL), guiding the agent's behavior. To express non-Markovian rewards, formalisms such as reward machines have been introduced to capture dependencies on histories. However, traditional reward machines lack the ability to model precise timing constraints, limiting their use in time-sensitive applications. In this paper, we propose timed reward machines (TRMs), which are an extension of reward machines that incorporate timing constraints into the reward structure. TRMs enable more expressive specifications with tunable reward logic, for example, imposing costs for delays and granting rewards for timely actions. We study model-free RL frameworks (i.e., tabular Q-learning) for learning optimal policies with TRMs under digital and real-time semantics. Our algorithms integrate the TRM into learning via abstractions of timed automata, and employ counterfactual-imagining heuristics that exploit the structure of the TRM to improve the search. Experimentally, we demonstrate that our algorithm learns policies that achieve high rewards while satisfying the timing constraints specified by the TRM on popular RL benchmarks. Moreover, we conduct comparative studies of performance under different TRM semantics, along with ablations that highlight the benefits of counterfactual-imagining.

About Time: Model-free Reinforcement Learning with Timed Reward Machines

TL;DR

The paper introduces Timed Reward Machines (TRMs) to express timing-sensitive, history-dependent rewards in reinforcement learning. It develops model-free Q-learning algorithms by constructing cross-product MDPs that combine the environment with TRMs under digital and real-time semantics, leveraging region/corner abstractions and counterfactual-imagining to manage timing constraints. Empirical results on Taxi and Frozen Lake show that TRMs improve learning when timing matters and that corner-point abstractions often outperform naive discretizations. The work lays groundwork for time-aware RL and points to future extensions to continuous-time models and priced zones for richer timing specifications.

Abstract

Reward specification plays a central role in reinforcement learning (RL), guiding the agent's behavior. To express non-Markovian rewards, formalisms such as reward machines have been introduced to capture dependencies on histories. However, traditional reward machines lack the ability to model precise timing constraints, limiting their use in time-sensitive applications. In this paper, we propose timed reward machines (TRMs), which are an extension of reward machines that incorporate timing constraints into the reward structure. TRMs enable more expressive specifications with tunable reward logic, for example, imposing costs for delays and granting rewards for timely actions. We study model-free RL frameworks (i.e., tabular Q-learning) for learning optimal policies with TRMs under digital and real-time semantics. Our algorithms integrate the TRM into learning via abstractions of timed automata, and employ counterfactual-imagining heuristics that exploit the structure of the TRM to improve the search. Experimentally, we demonstrate that our algorithm learns policies that achieve high rewards while satisfying the timing constraints specified by the TRM on popular RL benchmarks. Moreover, we conduct comparative studies of performance under different TRM semantics, along with ablations that highlight the benefits of counterfactual-imagining.

Paper Structure

This paper contains 24 sections, 8 theorems, 11 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. RL with Timed Specifications.
  4. RL with Non-Markovian Rewards.
  5. Timed Automata in Control and Planning.
  6. Preliminaries and Background
  7. Markov Decision Process
  8. Reinforcement Learning with Q-learning
  9. Problem formulation
  10. Timed Reward Machine (TRM)
  11. Interpretation of TRM on MDP
  12. Properties of TRM.
  13. The Digital Clock Setting
  14. Cross-product space.
  15. Q-learning on cross-product space.
  16. ...and 9 more sections

Key Result

Lemma 1

Let $\zeta= s_0\cdot (d_0, a_0)\cdots (d_{n},a_{n}) \cdot s_{n+1}$ be a trajectory and $\overline{\zeta} = s_0\cdot (\overline{d}_0, a_0)\cdots (\overline{d}_{n},a_{n}) \cdot s_{n+1}$ be its delay-bounded trajectory. Also, let $\mathcal{A}^\zeta: (u_0, v_0) \xrightarrow{d_0+1,\theta_0,r_0} \ldots \x

Figures (8)

  • Figure 1: An illustration of TRM on Gym Taxi domain: (a) Taxi Domain example, with a passenger in location red and destination in location blue, (b)A TRM that instructs the taxi to pick up a passenger and drop her at a destination, while moving slow, and (c) Rewards obtained using digital and real-time TRM, and reward machine.
  • Figure 2: Environment (above) along with TRM objective (below). The cost function $c$ is depicted in the top right corner of each state.
  • Figure 3: Trajectories $\zeta_1$ and $\zeta_2$ with induced words, TRM runs, and discounted rewards for digital and real-time settings ($\gamma=0.9$).
  • Figure 4: TRM (on the left) and MDP (on the right) illustrating agent behavior in the real-time setting.
  • Figure 5: Gym environments used in experiments.
  • ...and 3 more figures

Theorems & Definitions (13)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 3
  • proof
  • Lemma 4
  • ...and 3 more