Reinforcement Learning with Stochastic Reward Machines

TL;DR

This work addresses RL with non-Markovian, noisy rewards by introducing stochastic reward machines (SRMs) and a constraint-based learner, SRMI, that infers SRMs from exploration traces. SRMI enforces ε_c-consistency and uses constraint solving to produce minimal SRMs equivalent in expectation to the true environment, guaranteeing convergence to an optimal policy. The approach outperforms baselines and naive averaging on noisy case studies, while remaining effective in non-noisy settings and compatible with existing RL algorithms. The results substantiate SRMs as a practical framework for robust RL in temporally extended, stochastic reward scenarios, with broad implications for interpretable and reliable policy learning.

Abstract

Reward machines are an established tool for dealing with reinforcement learning problems in which rewards are sparse and depend on complex sequences of actions. However, existing algorithms for learning reward machines assume an overly idealized setting where rewards have to be free of noise. To overcome this practical limitation, we introduce a novel type of reward machines, called stochastic reward machines, and an algorithm for learning them. Our algorithm, based on constraint solving, learns minimal stochastic reward machines from the explorations of a reinforcement learning agent. This algorithm can easily be paired with existing reinforcement learning algorithms for reward machines and guarantees to converge to an optimal policy in the limit. We demonstrate the effectiveness of our algorithm in two case studies and show that it outperforms both existing methods and a naive approach for handling noisy reward functions.

Figures (6)

• Figure 1: A simplified example of the Mining environment grid and a trajectory. The agent's initial state is shown as a circle. Cells display their state labels, or are blank if the label is $\emptyset$. The trajectory indicated by arrows shows the agent collecting the equipment (E), finding platinum (P) and bringing it to the marketplace (M).
• Figure 2: Classical reward machine for the mining example (left) and stochastic reward machine (right). Transitions are labeled with input-output pairs, where labels are given by propositional formulas encoding subsets of $2^P$. All states have a transition to ${\mathsf{v}_T}$ with output $0$ on reading T (trap) (only depicted for $\mathsf{v}_1$). All remaining, missing transitions are self-loops with output $0$.
• Figure 3: Results on the mining environment
• Figure 4: Harvest environment
• Figure 5: Results on the harvest environment

Theorems & Definitions (22)

• definition thmcounterdefinition
• definition thmcounterdefinition
• definition thmcounterdefinition
• lemma thmcounterlemma
• definition thmcounterdefinition
• theorem thmcountertheorem
• corollary thmcountercorollary
• proof
• definition thmcounterdefinition
• lemma thmcounterlemma