Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation

TL;DR

This work addresses the safety and data-quality challenges of deploying online decision-making algorithms in digital health interventions. It offers a concrete set of monitoring guidelines, including a three-tier severity taxonomy and pre-defined fallback methods, to detect, triage, and mitigate issues in real time. Through two clinical case studies, Oralytics and MiWaves, the authors demonstrate how automated monitoring can detect problems such as out-of-memory, timeouts, and data-communication failures, and how fallbacks prevent harm and data loss. The guidelines provide a reusable blueprint for teams seeking to responsibly incorporate online RL into digital interventions and can extend to other online learning components in healthcare technology.

Abstract

Online AI decision-making algorithms are increasingly used by digital interventions to dynamically personalize treatment to individuals. These algorithms determine, in real-time, the delivery of treatment based on accruing data. The objective of this paper is to provide guidelines for enabling effective monitoring of online decision-making algorithms with the goal of (1) safeguarding individuals and (2) ensuring data quality. We elucidate guidelines and discuss our experience in monitoring online decision-making algorithms in two digital intervention clinical trials (Oralytics and MiWaves). Our guidelines include (1) developing fallback methods, pre-specified procedures executed when an issue occurs, and (2) identifying potential issues categorizing them by severity (red, yellow, and green). Across both trials, the monitoring systems detected real-time issues such as out-of-memory issues, database timeout, and failed communication with an external source. Fallback methods prevented participants from not receiving any treatment during the trial and also prevented the use of incorrect data in statistical analyses. These trials provide case studies for how health scientists can build monitoring systems for their digital intervention. Without these algorithm monitoring systems, critical issues would have gone undetected and unresolved. Instead, these monitoring systems safeguarded participants and ensured the quality of the resulting data for updating the intervention and facilitating scientific discovery. These monitoring guidelines and findings give digital intervention teams the confidence to include online decision-making algorithms in digital interventions.

Figures (1)

• Figure 1: Overview of the software components in the intervention system for both case studies. The backend controller acts as the central coordinator and interacts with the mobile app, the RL algorithm, and external systems (i.e. third party services not controlled by the development team - like proprietary toothbrush sensors). The monitoring dashboard pulls data from the backend controller and the RL algorithm for the digital intervention staff to monitor their performance and operational status in real-time. Note that the backend controller and the RL algorithm both have their individual databases to save necessary data to reproduce decisions and facilitate subsequent statistical analyses.