Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Numeric Reward Machines

Kristina Levina, Nikolaos Pappas, Athanasios Karapantelakis, Aneta Vulgarakis Feljan, Jendrik Seipp

TL;DR

This work addresses the limitation of reward machines that accept only Boolean features by introducing numeric features into RM-based reinforcement learning. It presents two extended RM variants—numeric--Boolean RM and numeric RM—and evaluates them in the Craft domain, showing faster learning than baseline Boolean RM approaches. The study provides theoretical shortest-path guarantees for single-target tasks and demonstrates empirically that numeric RM variants speed up learning for sequential object-visit tasks, revealing the practical potential of RM-guided RL in inherently numeric domains. Overall, incorporating numeric features broadens the applicability of reward machines to numeric tasks such as distance-based guidance and energy optimization, with significant implications for sample efficiency in RL.

Abstract

Reward machines inform reinforcement learning agents about the reward structure of the environment and often drastically speed up the learning process. However, reward machines only accept Boolean features such as robot-reached-gold. Consequently, many inherently numeric tasks cannot profit from the guidance offered by reward machines. To address this gap, we aim to extend reward machines with numeric features such as distance-to-gold. For this, we present two types of reward machines: numeric-Boolean and numeric. In a numeric-Boolean reward machine, distance-to-gold is emulated by two Boolean features distance-to-gold-decreased and robot-reached-gold. In a numeric reward machine, distance-to-gold is used directly alongside the Boolean feature robot-reached-gold. We compare our new approaches to a baseline reward machine in the Craft domain, where the numeric feature is the agent-to-target distance. We use cross-product Q-learning, Q-learning with counter-factual experiences, and the options framework for learning. Our experimental results show that our new approaches significantly outperform the baseline approach. Extending reward machines with numeric features opens up new possibilities of using reward machines in inherently numeric tasks.

Numeric Reward Machines

TL;DR

This work addresses the limitation of reward machines that accept only Boolean features by introducing numeric features into RM-based reinforcement learning. It presents two extended RM variants—numeric--Boolean RM and numeric RM—and evaluates them in the Craft domain, showing faster learning than baseline Boolean RM approaches. The study provides theoretical shortest-path guarantees for single-target tasks and demonstrates empirically that numeric RM variants speed up learning for sequential object-visit tasks, revealing the practical potential of RM-guided RL in inherently numeric domains. Overall, incorporating numeric features broadens the applicability of reward machines to numeric tasks such as distance-based guidance and energy optimization, with significant implications for sample efficiency in RL.

Abstract

Reward machines inform reinforcement learning agents about the reward structure of the environment and often drastically speed up the learning process. However, reward machines only accept Boolean features such as robot-reached-gold. Consequently, many inherently numeric tasks cannot profit from the guidance offered by reward machines. To address this gap, we aim to extend reward machines with numeric features such as distance-to-gold. For this, we present two types of reward machines: numeric-Boolean and numeric. In a numeric-Boolean reward machine, distance-to-gold is emulated by two Boolean features distance-to-gold-decreased and robot-reached-gold. In a numeric reward machine, distance-to-gold is used directly alongside the Boolean feature robot-reached-gold. We compare our new approaches to a baseline reward machine in the Craft domain, where the numeric feature is the agent-to-target distance. We use cross-product Q-learning, Q-learning with counter-factual experiences, and the options framework for learning. Our experimental results show that our new approaches significantly outperform the baseline approach. Extending reward machines with numeric features opens up new possibilities of using reward machines in inherently numeric tasks.
Paper Structure (17 sections, 18 equations, 8 figures)

This paper contains 17 sections, 18 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning
  4. Reward Machines
  5. Numeric Reward Machines
  6. Environment
  7. Boolean Reward Machines
  8. Numeric--Boolean Reward Machines
  9. Numeric Reward Machines
  10. Shortest-Path Guarantees
  11. Boolean Reward Machines
  12. Numeric--Boolean Reward Machines
  13. Numeric Reward Machines
  14. Experimental Evaluation
  15. Results and Discussion
  16. ...and 2 more sections

Figures (8)

  • Figure 1: Example grid map inspired by the Craft domain andreas2017modular. It contains objects a, b, and c.
  • Figure 2: Boolean reward machine with Boolean features a, b, and c for a sequential task a-b-c.
  • Figure 3: Numeric--Boolean reward machine with Boolean features $\space\raisebox{.15ex}{\footnotesize$\downarrow$} d_\texttt{a}$, $\space\raisebox{.15ex}{\footnotesize$\downarrow$} d_\texttt{b}$, $\space\raisebox{.15ex}{\footnotesize$\downarrow$} d_\texttt{c}$, $d_\texttt{a}$=$0$, $d_\texttt{b}$=$0$, and $d_\texttt{c}$=$0$ for a sequential task a-b-c.
  • Figure 4: Numeric reward machine with numeric features $d_\texttt{a}$, $d_\texttt{b}$, and $d_\texttt{c}$ and Boolean features $d_\texttt{a}$=$0$, $d_\texttt{b}$=$0$, and $d_\texttt{c}$=$0$ for a sequential task a-b-c.
  • Figure 5: Agent A can choose to approach either a$_1$ or a$_2$.
  • ...and 3 more figures