Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions

Nina Deliu, Joseph Jay Williams, Bibhas Chakraborty

TL;DR

This work provides the first unified technical survey on RL methods, complemented with case studies, for constructing various types of AIs in healthcare, and bridges two seemingly different AI domains, dynamic treatment regimes and just‐in‐time adaptive interventions in mobile health.

Abstract

In recent years, reinforcement learning (RL) has acquired a prominent position in health-related sequential decision-making problems, gaining traction as a valuable tool for delivering adaptive interventions (AIs). However, in part due to a poor synergy between the methodological and the applied communities, its real-life application is still limited and its potential is still to be realized. To address this gap, our work provides the first unified technical survey on RL methods, complemented with case studies, for constructing various types of AIs in healthcare. In particular, using the common methodological umbrella of RL, we bridge two seemingly different AI domains, dynamic treatment regimes and just-in-time adaptive interventions in mobile health, highlighting similarities and differences between them and discussing the implications of using RL. Open problems and considerations for future research directions are outlined. Finally, we leverage our experience in designing case studies in both areas to showcase the significant collaborative opportunities between statistical, RL, and healthcare researchers in advancing AIs.

Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions

TL;DR

This work provides the first unified technical survey on RL methods, complemented with case studies, for constructing various types of AIs in healthcare, and bridges two seemingly different AI domains, dynamic treatment regimes and just‐in‐time adaptive interventions in mobile health.

Abstract

In recent years, reinforcement learning (RL) has acquired a prominent position in health-related sequential decision-making problems, gaining traction as a valuable tool for delivering adaptive interventions (AIs). However, in part due to a poor synergy between the methodological and the applied communities, its real-life application is still limited and its potential is still to be realized. To address this gap, our work provides the first unified technical survey on RL methods, complemented with case studies, for constructing various types of AIs in healthcare. In particular, using the common methodological umbrella of RL, we bridge two seemingly different AI domains, dynamic treatment regimes and just-in-time adaptive interventions in mobile health, highlighting similarities and differences between them and discussing the implications of using RL. Open problems and considerations for future research directions are outlined. Finally, we leverage our experience in designing case studies in both areas to showcase the significant collaborative opportunities between statistical, RL, and healthcare researchers in advancing AIs.
Paper Structure (31 sections, 38 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 31 sections, 38 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Adaptive Interventions in Healthcare
  3. Dynamic treatment regimes
  4. Just-in-time adaptive interventions in mHealth
  5. The Reinforcement Learning Framework
  6. A preliminary note: causal inference and RL
  7. Formalization of the general RL problem
  8. Online and offline RL
  9. Formalization of specific RL problems
  10. Markov decision processes
  11. Multi-armed bandits
  12. RL and AIs: a joint overview
  13. A Survey of RL Methods for Adaptive Interventions
  14. RL methods for dynamic treatment regimes
  15. A historical overview
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Simplified schematic of a two-stage AI and its key components, inspired by the weight loss management study in pfammatter_smart_2019
  • Figure 2: Schematic of the weight loss SMART in pfammatter_smart_2019
  • Figure 3: Schematic of the DIAMANTE micro-randomized trial aguilera2020mhealth
  • Figure 4: Graphical representation of the states, actions, and rewards relationship in a full-RL, MDP-based RL (MDP-RL), and stochastic (both contextual and context free) MAB. Solid and dashed lines indicate a direct and indirect (e.g., time-delayed) effect, respectively.
  • Figure 5: Schematic of a feed-forward neural network. It is characterized by a set of neurons, structured in four layers ($L=4$), where each neuron processes the information forward from one layer to the next one. Information is nonlinearly transformed according to unknown weights $W^{(l)}$ and bias $b^{(l)}$ parameters, $l=1,\dots,L-1$.
  • ...and 1 more figures