Generalization in Monitored Markov Decision Processes (Mon-MDPs)
Montaser Mohammedalamen, Michael Bowling
TL;DR
The paper tackles reward observability in reinforcement learning by extending Mon-MDPs to non-tabular settings with function approximation. By jointly training a neural reward model and a Q-model, the approach enables generalization from monitored states with observable rewards to unmonitored states with unobservable rewards, achieving near-optimal performance in some unsolvable Mon-MDPs. However, the authors identify overgeneralization as a critical risk where rewards are incorrectly extrapolated to novel states. To mitigate this, they adopt robust policy optimization using reward uncertainty and CVaR (via $k$-of-$N$ CFR), yielding cautious policies that balance leveraging familiar rewards with avoiding unsafe extrapolations. The work advances Mon-MDPs toward real-world applicability, highlighting both the potential for improved generalization and the need for uncertainty-aware safeguards.
Abstract
Reinforcement learning (RL) typically models the interaction between the agent and environment as a Markov decision process (MDP), where the rewards that guide the agent's behavior are always observable. However, in many real-world scenarios, rewards are not always observable, which can be modeled as a monitored Markov decision process (Mon-MDP). Prior work on Mon-MDPs have been limited to simple, tabular cases, restricting their applicability to real-world problems. This work explores Mon-MDPs using function approximation (FA) and investigates the challenges involved. We show that combining function approximation with a learned reward model enables agents to generalize from monitored states with observable rewards, to unmonitored environment states with unobservable rewards. Therefore, we demonstrate that such generalization with a reward model achieves near-optimal policies in environments formally defined as unsolvable. However, we identify a critical limitation of such function approximation, where agents incorrectly extrapolate rewards due to overgeneralization, resulting in undesirable behaviors. To mitigate overgeneralization, we propose a cautious police optimization method leveraging reward uncertainty. This work serves as a step towards bridging this gap between Mon-MDP theory and real-world applications.