Policy Optimization for Personalized Interventions in Behavioral Health

TL;DR

A new approach to optimizing personalized interventions for patients to maximize a long-term outcome, in which interventions are costly and capacity constrained, is presented, which decomposes the state space for a system of patients to the individual level and then approximates one step of policy iteration.

Abstract

Behavioral health interventions, delivered through digital platforms, have the potential to significantly improve health outcomes, through education, motivation, reminders, and outreach. We study the problem of optimizing personalized interventions for patients to maximize a long-term outcome, where interventions are costly and capacity-constrained. We assume we have access to a historical dataset collected from an initial pilot study. We present a new approach for this problem that we dub DecompPI, which decomposes the state space for a system of patients to the individual level and then approximates one step of policy iteration. Implementing DecompPI simply consists of a prediction task using the dataset, alleviating the need for online experimentation. DecompPI is a generic model-free algorithm that can be used irrespective of the underlying patient behavior model. We derive theoretical guarantees on a simple, special case of the model that is representative of our problem setting. When the initial policy used to collect the data is randomized, we establish an approximation guarantee for DecompPI with respect to the improvement beyond a null policy that does not allocate interventions. We show that this guarantee is robust to estimation errors. We then conduct a rigorous empirical case study using real-world data from a mobile health platform for improving treatment adherence for tuberculosis. Using a validated simulation model, we demonstrate that DecompPI can provide the same efficacy as the status quo approach with approximately half the capacity of interventions. DecompPI is simple and easy to implement for an organization aiming to improve long-term behavior through targeted interventions, and this paper demonstrates its strong performance both theoretically and empirically, particularly in resource-limited settings.

Figures (7)

• Figure 1: Average overall verification rate over 50 runs for each policy and budget. The shaded region indicates a 95% confidence interval. The star represents the operating point for Keheala.
• Figure 2: MDP for patient $i$.
• Figure 3: Calibration plots for $f(S, 0)$ and $f(S, 1)$ for simulation validation. We group the samples based on the simulated probability of verification into bins with a 10% range, which we label by the lower number. For example, the 0.3 bin on the x-axis represents the samples whose probability of verification according to $f$ is in $[0.3, 0.4)$; hence we should expect the actual number of verifications of those samples to be close to 0.35.
• Figure 4: Average overall verification rate over 50 runs for each policy and budget. The overall verification rate for the $\mathsf{NULL}$ policy was 54.2%. The shaded region indicates a 95% confidence interval. The star represents the operating point for Keheala.
• Figure 5: Differences in the distribution of patient verification rates compared to $\mathsf{Baseline}$. The bins represent the difference in the number of patients whose overall verification rate is between $0-10\%$, $10-20\%, \dots, 90-100\%$. For example, the first bin in (a) shows that there were 28 fewer patients whose verification rate was between 0 and 10% under $\mathsf{DecompPI}$, compared to $\mathsf{Baseline}$. There were 3594 patients in total, and the budget was fixed at 26.

Theorems & Definitions (21)

• Proposition 1
• Theorem 1
• Proposition 2
• Theorem 2: Main Result
• Corollary 1: Weaker result
• Theorem 3
• Proposition A1
• Proposition A2
• Proposition A3
• Proposition A4