Criticality and Safety Margins for Reinforcement Learning

TL;DR

The paper defines true criticality as the expected drop in discounted reward when an RL agent's policy is perturbed by random actions for $n$ steps, and introduces proxy criticality as a fast, correlate-to-true-metric suitable for real-time use. A data-driven safety-margin pipeline maps proxy values to probabilistic bounds on true criticality, enabling actionable oversight by identifying how many consecutive mistakes can be tolerated before performance degrades beyond a tolerance $\zeta$. The framework is validated on Pong and Beamrider with APE-X and A3C, showing that proxy-to-safety-margin relationships are strong enough to guide supervision, and that a small fraction of the most critical moments often accounts for a large share of losses. The approach offers practical benefits for debugging and monitoring autonomous agents in safety-critical settings, while highlighting the need for diverse proxy metrics and robustness to policy imperfections. Overall, the paper provides a principled, interpretable mechanism to quantify and bound the risk of bad decisions in reinforcement learning systems.

Abstract

State of the art reinforcement learning methods sometimes encounter unsafe situations. Identifying when these situations occur is of interest both for post-hoc analysis and during deployment, where it might be advantageous to call out to a human overseer for help. Efforts to gauge the criticality of different points in time have been developed, but their accuracy is not well established due to a lack of ground truth, and they are not designed to be easily interpretable by end users. Therefore, we seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users. We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions. We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality. Safety margins make these interpretable, when defined as the number of random actions for which performance loss will not exceed some tolerance with high confidence. We demonstrate this approach in several environment-agent combinations; for an A3C agent in an Atari Beamrider environment, the lowest 5% of safety margins contain 47% of agent losses; i.e., supervising only 5% of decisions could potentially prevent roughly half of an agent's errors. This criticality framework measures the potential impacts of bad decisions, even before those decisions are made, allowing for more effective debugging and oversight of autonomous agents.

Figures (10)

• Figure 1: Normalized KDE plots capturing the relationship between proxy criticality (on the horizontal axis) and estimated true criticality (on the vertical axis), with contours and trendlines, for an A3C model trained on Beamrider. A portion of this figure appeared in grushin2023, with some differences in notation.
• Figure 2: Normalized KDE plot excerpt for an A3C model trained on Pong; see \ref{['sec:app:results']} (\ref{['fig:results:kde:a3c:pong']}) for the full figure.
• Figure 3: Heatmaps displaying safety margins for Beamrider in the proxy criticality metric vs. tolerance space, for APE-X (left) and A3C (right). This figure was originally presented in grushin2023, albeit with some differences in notation.
• Figure 4: Histograms of metric values $p_{m}(t_m)$ for Beamrider, with vs. without uniform sampling, for APE-X (left) and A3C (right); 50 equal-sized bins were used. The shaded region (on the right of each plot) corresponds to the top $5\%$ of $p_{m}(t_m)$ values, which were removed.
• Figure 5: An illustration of the time step selection approach. Here, three episodes have been processed so far; in these episodes, time steps $18$, $10$ and $6$ were selected, with criticality values $p_1(18)$, $p_2(10)$ and $p_3(6)$, respectively (represented via blue circles, with the horizontal position of a circle corresponding to its criticality value, where higher values appear farther to the right). For episode four, suppose that there are only two time steps $t_{1}$ and $t_{2}$ (this is done for ease of illustration; note that for the environments that we consider, an episode will often have thousands of time steps); these have criticality values $p_4(1)$ and $p_4(2)$, and are represented via red circles. Because $p_4(1)$ is closer than $p_4(2)$ to a criticality value for a previously-selected time step (specifically, $p_4(1)$ is only slightly higher than $p_2(10)$), time step $2$ is selected, and the criticality value $p_4(2)$ is added to the set of tracked criticality values (represented by the grey box). Note that it is not necessary to store the previously-selected time steps, only the criticality values at these time steps.