Testing Stationarity and Change Point Detection in Reinforcement Learning

TL;DR

This work tackles nonstationarity in offline reinforcement learning by developing a statistically principled test for the stationarity of the optimal Q-function $Q^{opt}$ using pre-collected data. It introduces three model-free test statistics based on estimated Q-functions, a multiplier bootstrap for critical values, and a change-point detection procedure to identify the most recent stationary segment for online policy learning. The authors prove consistency and derive finite-sample properties, validate performance via simulations and a real-world Intern Health Study application, and provide a Python implementation. The approach enables robust policy optimization in nonstationary environments and can be integrated with existing RL methods to adapt to evolving dynamics in domains like mHealth and healthcare monitoring.

Abstract

We consider offline reinforcement learning (RL) methods in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the system transition and the reward function to be constant over time. However, the stationarity assumption is restrictive in practice and is likely to be violated in a number of applications, including traffic signal control, robotics and mobile health. In this paper, we develop a consistent procedure to test the nonstationarity of the optimal Q-function based on pre-collected historical data, without additional online data collection. Based on the proposed test, we further develop a sequential change point detection method that can be naturally coupled with existing state-of-the-art RL methods for policy optimization in nonstationary environments. The usefulness of our method is illustrated by theoretical results, simulation studies, and a real data example from the 2018 Intern Health Study. A Python implementation of the proposed procedure is available at https://github.com/limengbinggz/CUSUM-RL.

Figures (13)

• Figure 3.1: Examples of $Q^{opt}$ at a given state-action pair, with an abrupt change point (left panel) and a gradual change point (right panel) at $t=50$. $T_0=0$ in both examples.
• Figure 5.1: Empirical type-I errors and powers of the proposed test and their associated 95% confidence intervals under settings described in Section \ref{['sec:sim:1d']}, with $N=25$. Abbreviations: Homo for homogeneous, PC for piecewise constant, and Sm for smooth.
• Figure 5.2: Distribution of detected change points under simulation settings in Section \ref{['sec:sim:1d']} with $N=25$.
• Figure 5.3: Distribution of the difference between the average value $(T_{end}-T)^{-1}\sum_{t=T+1}^{T_{end}} {\mathbb{E}} R_t$ under the proposed policy and those under policies computed by other baseline methods, under settings in Section \ref{['sec:sim:1d']} with strong signal-to-noise ratio. The proposed policy is based on the change point detected by the $\ell_1$-type test \ref{['eqn:teststat1']}. In all scenarios, we find the normalized or unnormalized tests \ref{['eqn:teststatinf']} and \ref{['eqn:teststatninf']} yield similar average values.
• Figure 6.1: $p$-values over different values of $\kappa$ (the number of time points from the last time point $T$) under $\gamma=0.9$ (top) and $0.95$ (bottom) among the three specialties considered in IHS.

Theorems & Definitions (21)

• Remark 2.1
• Remark 2.2
• Theorem 2.3: Stationarity relationships
• Remark 2.4
• Remark 2.5
• Remark 3.1
• Remark 3.2
• Remark 3.3
• Remark 3.4
• Remark 4.1