Learning to Be Cautious
Montaser Mohammedalamen, Dustin Morrill, Alexander Sieusahai, Yash Satsangi, Michael Bowling
TL;DR
The paper tackles safe, cautious behavior in RL under novel, unseen states by combining an ensemble of neural reward models to capture epistemic uncertainty with a $k$-of-$N$ CFR robust optimization to produce cautious policies without task-specific safety tuning. It formalizes the extension of $k$-of-$N$ CFR to continuing MDPs and proves regret bounds, while empirically validating cautious behavior on MNIST-based tasks and a driving gridworld. The approach demonstrates that caution can be learned autonomously and scales with robustness settings, preserving performance on familiar tasks while increasing cautious actions in novel settings. This work offers a practical pathway to safer deployed AI systems by integrating uncertainty-aware robustness into reinforcement learning, though it notes limitations and directions for scaling to larger, more complex environments.
Abstract
A key challenge in the field of reinforcement learning is to develop agents that behave cautiously in novel situations. It is generally impossible to anticipate all situations that an autonomous system may face or what behavior would best avoid bad outcomes. An agent that can learn to be cautious would overcome this challenge by discovering for itself when and how to behave cautiously. In contrast, current approaches typically embed task-specific safety information or explicit cautious behaviors into the system, which is error-prone and imposes extra burdens on practitioners. In this paper, we present both a sequence of tasks where cautious behavior becomes increasingly non-obvious, as well as an algorithm to demonstrate that it is possible for a system to learn to be cautious. The essential features of our algorithm are that it characterizes reward function uncertainty without task-specific safety information and uses this uncertainty to construct a robust policy. Specifically, we construct robust policies with a k-of-N counterfactual regret minimization (CFR) subroutine given learned reward function uncertainty represented by a neural network ensemble. These policies exhibit caution in each of our tasks without any task-specific safety tuning. Our code is available at https://github.com/montaserFath/Learning-to-be-Cautious