Expressive Reward Synthesis with the Runtime Monitoring Language
Daniel Donnelly, Angelo Ferrando, Francesco Belardinelli
TL;DR
RML Reward Machines extend traditional Reward Machines by integrating the Runtime Monitoring Language to support memory and parametric event handling, enabling non-regular reward specifications in reinforcement learning. They expose monitor states to the agent to make rewards deterministic and provide intermediate rewards to shape learning, demonstrated on counting-like tasks such as $A^{N} B C D^{N}$; experiments show improved task specification and learning speed relative to Counting Reward Automata and RMLGym. The approach yields constant-branch specification for many conditional tasks, and generalizes standard RM expressivity to non-regular languages, offering a practical path toward memory-aware reward design with potential impact on robotic navigation and safety-critical RL. Overall, RML Reward Machines provide a flexible, expressive framework for designing and learning memory-dependent rewards in RL, with strong empirical evidence of advantages in complex event handling and counting tasks.
Abstract
A key challenge in reinforcement learning (RL) is reward (mis)specification, whereby imprecisely defined reward functions can result in unintended, possibly harmful, behaviours. Indeed, reward functions in RL are typically treated as black-box mappings from state-action pairs to scalar values. While effective in many settings, this approach provides no information about why rewards are given, which can hinder learning and interpretability. Reward Machines address this issue by representing reward functions as finite state automata, enabling the specification of structured, non-Markovian reward functions. However, their expressivity is typically bounded by regular languages, leaving them unable to capture more complex behaviours such as counting or parametrised conditions. In this work, we build on the Runtime Monitoring Language (RML) to develop a novel class of language-based Reward Machines. By leveraging the built-in memory of RML, our approach can specify reward functions for non-regular, non-Markovian tasks. We demonstrate the expressiveness of our approach through experiments, highlighting additional advantages in flexible event-handling and task specification over existing Reward Machine-based methods.