StepCountJITAI: simulation environment for RL with application to physical activity adaptive intervention

TL;DR

StepCountJITAI is introduced, an RL environment designed to foster research on RL methods that address the significant challenges of policy learning for adaptive behavioral interventions.

Abstract

The use of reinforcement learning (RL) to learn policies for just-in-time adaptive interventions (JITAIs) is of significant interest in many behavioral intervention domains including improving levels of physical activity. In a messaging-based physical activity JITAI, a mobile health app is typically used to send messages to a participant to encourage engagement in physical activity. In this setting, RL methods can be used to learn what intervention options to provide to a participant in different contexts. However, deploying RL methods in real physical activity adaptive interventions comes with challenges: the cost and time constraints of real intervention studies result in limited data to learn adaptive intervention policies. Further, commonly used RL simulation environments have dynamics that are of limited relevance to physical activity adaptive interventions and thus shed little light on what RL methods may be optimal for this challenging application domain. In this paper, we introduce StepCountJITAI, an RL environment designed to foster research on RL methods that address the significant challenges of policy learning for adaptive behavioral interventions.

Figures (17)

• Figure 1: Overview of StepCountJITAI in an RL loop. StepCountJITAI is a simulation environment for physical activity adaptive interventions. StepCountJITAI models stochastic behavioral dynamics and context uncertainty, with parameters to control stochasticity. Step count is used as the reward. The actions correspond to physical activity motivational messages with different contextualization levels. The messages can be non-contextualized, or customized to a binary context. The behavioral variables are: habitation and disengagement risk.
• Figure 2: Histograms for stochastic ${h}_t$, ${d}_t$, ${s}_t$ using $a_{hd}=0.5$, $\sigma_s=10$.
• Figure 3: Histograms for stochastic ${h}_t$, ${d}_t$, ${s}_t$ using $a_{hd}=0.2$, $\sigma_s=2$.
• Figure 4: Histograms for stochastic ${h}_t$, ${d}_t$, ${s}_t$ using $\kappa_h=200$, $\kappa_d=200$, $\sigma_s=10$.
• Figure 5: Histograms for stochastic ${h}_t$, ${d}_t$, ${s}_t$ using $\kappa_h=1000$, $\kappa_d=1000$, $\sigma_s=2$.