Cumulative Treatment Effect Testing under Continuous Time Reinforcement Learning
Jiuchen Zhang, Annie Qu
TL;DR
We present a continuous-time reinforcement learning framework to test cumulative treatment effects, incorporating carryover via the infinite-horizon value function and defining the average treatment effect (ATE) as $\tau=\int_{\mathcal{S}}(V_1(s)-V_0(s))\mathbb{G}(ds)$. The methodology derives a Bellman-based estimating equation, identifies $V_0$ and $V_1$ under CA, CTSR, MA, and CMIA, and constructs a plug-in estimator $\widehat{\tau}$ with a robust sandwich variance to form the test statistic $Z=\sqrt{nI}\,\widehat{\tau}/\widehat{\sigma}$, which is asymptotically $N(0,1)$. The framework accommodates irregular, multi-resolution observations by using a drift-based approximation to the infinitesimal generator and a flexible basis for $V_a$, enabling accurate detection of both immediate and carryover effects. Simulations and a real data study on the OhioT1DM dataset demonstrate superior power relative to discrete-time methods (t-test, DML, SAVE) while preserving Type I error, highlighting the method’s practical utility for dynamic, personalized interventions in mobile-health and related domains. Potential extensions include online monitoring and stochastic-differential equation formulations to model diffusion, broadening applicability to real-time adaptive decision making.
Abstract
Understanding the impact of treatment effect over time is a fundamental aspect of many scientific and medical studies. In this paper, we introduce a novel approach under a continuous-time reinforcement learning framework for testing a treatment effect. Specifically, our method provides an effective test on carryover effects of treatment over time utilizing the average treatment effect (ATE). The average treatment effect is defined as difference of value functions over an infinite horizon, which accounts for cumulative treatment effects, both immediate and carryover. The proposed method outperforms existing testing procedures such as discrete time reinforcement learning strategies in multi-resolution observation settings where observation times can be irregular. Another advantage of the proposed method is that it can capture treatment effects of a shorter duration and provide greater accuracy compared to discrete-time approximations, through the use of continuous-time estimation for the value function. We establish the asymptotic normality of the proposed test statistics and apply it to OhioT1DM diabetes data to evaluate the cumulative treatment effects of bolus insulin on patients' glucose levels.