Assessing the Impact of Distribution Shift on Reinforcement Learning Performance

TL;DR

The paper tackles the fragility of RL evaluation under distribution shift and the limitations of relying on point estimates. It proposes a time-series evaluation framework that integrates causal-inference tools, such as difference-in-differences and counterfactual analyses, with forecasting methods and prediction intervals to quantify post-shift impact. The authors demonstrate the approach through adversarial attacks on Atari agents (A2C and PPO) and multi-agent switching in PowerGridworld, illustrating how distribution shifts can erode performance and how time-series analysis reveals robustness differences. This work advances RL safety and regulation by providing a practical, test-time evaluation protocol applicable to both single- and multi-agent settings, aiming to improve reproducibility and deployment reliability.

Abstract

Research in machine learning is making progress in fixing its own reproducibility crisis. Reinforcement learning (RL), in particular, faces its own set of unique challenges. Comparison of point estimates, and plots that show successful convergence to the optimal policy during training, may obfuscate overfitting or dependence on the experimental setup. Although researchers in RL have proposed reliability metrics that account for uncertainty to better understand each algorithm's strengths and weaknesses, the recommendations of past work do not assume the presence of out-of-distribution observations. We propose a set of evaluation methods that measure the robustness of RL algorithms under distribution shifts. The tools presented here argue for the need to account for performance over time while the agent is acting in its environment. In particular, we recommend time series analysis as a method of observational RL evaluation. We also show that the unique properties of RL and simulated dynamic environments allow us to make stronger assumptions to justify the measurement of causal impact in our evaluations. We then apply these tools to single-agent and multi-agent environments to show the impact of introducing distribution shifts during test time. We present this methodology as a first step toward rigorous RL evaluation in the presence of distribution shifts.

Figures (24)

• Figure 1: In this simplified plot of agent performance in the presence of worsening distribution shifts over time. All three agents have average returns of 10. It is clear, however, that agent 3 is the least desired agent over time. Even though agent 3 starts out with the highest average returns, it seems to have overfit to the training environment and fails to maintain its superior performance. Point estimates alone would not capture this behavior.
• Figure 2: Left: The differences in performance are clear because the prediction intervals do not overlap at the end of the plot. Hence, Agent 1 has the best performance because the forecast is not decreasing and the prediction interval is small. Right: Here, all agents have noisier performance. Even though Agent 3 still has a downward trend, it briefly spikes up to match Agent 1's performance. Hence, we want prediction intervals that anticipate this uncertainty by showing interval overlap between agent performance over time. There is no longer a significant difference between Agents 2 and 3 because their prediction intervals overlap at every time step.
• Figure 3: The causal impact plots shown here illustrate the impact of FGSM adversarial attacks on RL agents trained on the Pong Atari game. Each row represents an Atari game. Each column represents a RL algorithm (A2C or PPO). The original plots here show the rolling mean of the rewards over time. The pointwise plots show the difference between the counterfactual performance and the performance when the agent is attacked. The cumulative performance is the summation of the rewards gained or lost over time. As expected, the performance tends to drop as $\epsilon$ increases.
• Figure 4: These plots take the rolling mean (window=25) of the observational performance up to a certain time point with some probability of being in the presence of adversarial attacks. After that time point, it shows the time series forecast with 99% prediction intervals. Top Row: In the PongNoFrameskip-v4 plots, PPO performs significantly better and has smaller prediction intervals. The plot on the right, however, uses attacks with higher $\epsilon$, where both agents have larger prediction intervals. Bottom Row: In the BreakoutNoFrameskip-v4 plots, the prediction intervals overlap. One noticeable difference is that the stronger attacks cause the PPO interval to shrink in size while the mean rewards are slightly less. This decrease in variability accounts for the decrease in the maximum rewards achieved.
• Figure 5: The plots here show the impact of the ad hoc switching of agents in a group of 5 in the PowerGridworld environment. Left Column: We replace 1, 2, or 3 agents out of the group of 5 with agents that trained with a different group. While there is little change when only replacing 1 or 2 agents, we see that performance dramatically decreases when 3 agents have been switched out. Right Column: We see that just switching out 1 agent in the group with 1 untrained agent causes a significantly large decrease in group performance.

Theorems & Definitions (2)

• Definition 1
• Definition 2