Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monitored Markov Decision Processes

Simone Parisi, Montaser Mohammedalamen, Alireza Kazemipour, Matthew E. Taylor, Michael Bowling

TL;DR

Monitored MDPs introduce a monitor as a separate Markov decision process that governs reward observability, addressing RL settings where environment rewards $r^\text{e}_t$ are not always observable and the agent receives proxy rewards $\hat{r}^\text{e}_t$ (possibly $\bot$). The paper formalizes the Mon-MDP framework, defines optimality with joint rewards $r_t = r^\text{e}_t + r^\text{m}_t$, and analyzes convergence under ergodicity and truthful monitors, arguing for the monitor as a distinct learning component. Through gridworld experiments, it compares multiple Q-learning variants and a reward-model approach, showing that naive handling of unobserved rewards can fail while learning a reward model restores convergence under suitable conditions. This work lays a foundation for further theory on convergence guarantees, exploration strategies, and practical algorithms for real-world tasks with imperfect or delayed reward feedback.

Abstract

In reinforcement learning (RL), an agent learns to perform a task by interacting with an environment and receiving feedback (a numerical reward) for its actions. However, the assumption that rewards are always observable is often not applicable in real-world problems. For example, the agent may need to ask a human to supervise its actions or activate a monitoring system to receive feedback. There may even be a period of time before rewards become observable, or a period of time after which rewards are no longer given. In other words, there are cases where the environment generates rewards in response to the agent's actions but the agent cannot observe them. In this paper, we formalize a novel but general RL framework - Monitored MDPs - where the agent cannot always observe rewards. We discuss the theoretical and practical consequences of this setting, show challenges raised even in toy environments, and propose algorithms to begin to tackle this novel setting. This paper introduces a powerful new formalism that encompasses both new and existing problems and lays the foundation for future research.

Monitored Markov Decision Processes

TL;DR

Monitored MDPs introduce a monitor as a separate Markov decision process that governs reward observability, addressing RL settings where environment rewards are not always observable and the agent receives proxy rewards (possibly ). The paper formalizes the Mon-MDP framework, defines optimality with joint rewards , and analyzes convergence under ergodicity and truthful monitors, arguing for the monitor as a distinct learning component. Through gridworld experiments, it compares multiple Q-learning variants and a reward-model approach, showing that naive handling of unobserved rewards can fail while learning a reward model restores convergence under suitable conditions. This work lays a foundation for further theory on convergence guarantees, exploration strategies, and practical algorithms for real-world tasks with imperfect or delayed reward feedback.

Abstract

In reinforcement learning (RL), an agent learns to perform a task by interacting with an environment and receiving feedback (a numerical reward) for its actions. However, the assumption that rewards are always observable is often not applicable in real-world problems. For example, the agent may need to ask a human to supervise its actions or activate a monitoring system to receive feedback. There may even be a period of time before rewards become observable, or a period of time after which rewards are no longer given. In other words, there are cases where the environment generates rewards in response to the agent's actions but the agent cannot observe them. In this paper, we formalize a novel but general RL framework - Monitored MDPs - where the agent cannot always observe rewards. We discuss the theoretical and practical consequences of this setting, show challenges raised even in toy environments, and propose algorithms to begin to tackle this novel setting. This paper introduces a powerful new formalism that encompasses both new and existing problems and lays the foundation for future research.
Paper Structure (36 sections, 3 theorems, 17 equations, 17 figures, 3 tables)

This paper contains 36 sections, 3 theorems, 17 equations, 17 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Monitored MDPs
  4. Why RL Needs Mon-MDPs
  5. Related Work
  6. Mon-MDPs Optimality
  7. Policy Optimality
  8. Convergence to an Optimal Policy
  9. Empirical Analysis of Mon-MDPs
  10. The Environment and The Monitors
  11. The Algorithms
  12. Algorithm 1: Assign $\boldsymbol{\bot = 0}$.
  13. Algorithm 2: Ignore $\boldsymbol{\hat{r}^\textsc{e}_t = \bot}$.
  14. Algorithm 3: Two Q-functions (Joint Greedy Policy).
  15. Algorithm 4: Two Q-functions (Sequential Greedy Policy).
  16. ...and 21 more sections

Key Result

Proposition 1

There exist an algorithm such that for any Mon-MDP with finite states and actions satisfying Properties def:ergodic_mmdp, def:env_ergodic_mon, and def:truthful_mon, the algorithm converges to an optimal policy of that Mon-MDP.

Figures (17)

  • Figure 1: Example of Monitored MDP. The agent is tasked with household chores but needs the owner or home sensors to observe rewards. If the owner is not home or the sensors are unavailable (left), the agent will not receive positive rewards for cleaning dishes or negative rewards for spilling water. Thus, the agent must learn how to seek monitoring --- where to move for sensory feedback or when the owner is home (center) --- and act appropriately even when monitoring is unavailable (e.g., be cautious when not being monitored). Eventually, the agent can judge actions on its own without any monitoring (right).
  • Figure 2: In classic MDPs (left), the agent directly observes environment rewards $r^\textsc{e}$. In Mon-MDPs (right), the agent cannot. Instead, it receives proxy rewards $\hat{r}^\textsc{e}$ via the monitor. Like the environment, the monitor is governed by a Markovian transition function and has its own rewards $r^\textsc{m}$. At every step, the agent observes the state of both the environment and the monitor, and executes actions affecting both. The agent's goal is to maximize the cumulative sum of rewards $(r^\textsc{e} + r^\textsc{m})$, while observing $\hat{r}^\textsc{e}$ instead of $r^\textsc{e}$. If the reward is unobservable, the agent receives $\hat{r}^\textsc{e} = \bot$.
  • Figure 3: Example showing why learning with unobservable reward is non-trivial. The agent moves between states A, B, and C, but rewards can be observed only with monitoring. Otherwise, the agent observes $\hat{r}^\textsc{e}_t = \bot$. If the agent interprets the lack of reward as meaning that all actions are equally good, it could believe that transitioning from B to A is as good as transitioning from B to C.
  • Figure 4: In our Mon-MDPs, the agent starts in the top-left cell and has to reach the goal avoiding penalty cell. There are nine states $s^\textsc{e}$ (one for each cell) and four actions $a^\textsc{e}$ ($\texttt{LEFT}, \texttt{RIGHT}, \texttt{UP}, \texttt{DOWN}$). Rewards $r^\textsc{e}$ are 1 (goal), -10 (penalty cells), and 0 (otherwise). Episodes end when the agent reaches the goal or after 50 steps. We propose three levels of difficulty depending on the presence of penalty cells and on the type of monitor.
  • Figure 5: If $\bot = 0$, the agent asks to be monitored only when rewards are positive, "ignoring" negative rewards.
  • ...and 12 more figures

Theorems & Definitions (9)

  • Proposition 1: Sufficient Conditions for Convergence to an Optimal Policy
  • Proof 1
  • Definition 1: Indistinguishability $\mathbb{I}$
  • Definition 2: Solvable Mon-MDP
  • Lemma 1
  • Definition 3: Hopeless Mon-MDP
  • Definition 4: Invariant Mon-MDP
  • Proposition 2: Sequential Algorithm Convergence in Invariant Mon-MDPs
  • Proof 2