Optimal Control of Mechanical Ventilators with Learned Respiratory Dynamics

TL;DR

This work frames the management of ventilators for patients with ARDS as a sequential decision making problem using the Markov decision process framework, and implements and compares controllers based on clinical guidelines contained in the ARDSnet protocol, optimal control theory, and learned latent dynamics represented as neural networks.

Abstract

Deciding on appropriate mechanical ventilator management strategies significantly impacts the health outcomes for patients with respiratory diseases. Acute Respiratory Distress Syndrome (ARDS) is one such disease that requires careful ventilator operation to be effectively treated. In this work, we frame the management of ventilators for patients with ARDS as a sequential decision making problem using the Markov decision process framework. We implement and compare controllers based on clinical guidelines contained in the ARDSnet protocol, optimal control theory, and learned latent dynamics represented as neural networks. The Pulse Physiology Engine's respiratory dynamics simulator is used to establish a repeatable benchmark, gather simulated data, and quantitatively compare these controllers. We score performance in terms of measured improvement in established ARDS health markers (pertaining to improved respiratory rate, oxygenation, and vital signs). Our results demonstrate that techniques leveraging neural networks and optimal control can automatically discover effective ventilation management strategies without access to explicit ventilator management procedures or guidelines (such as those defined in the ARDSnet protocol).

Figures (3)

• Figure 1: An overview of this work. A policy $\pi$ considers an ARDS patient's health state $\mathbf{s} \in \mathbb{R}^{27}$ and determines a ventilator action $\mathbf{a} \in \mathbb{R}^{6}$ through some decision-making algorithm. This process repeats until $48$ hours have been simulated (or $96$ sequential ventilator actions spaced at $\Delta t = 30$ minutes). The algorithms in this work (generally) operate by maximizing some reward $R(\mathbf{s},\mathbf{a})$, where a high-reward is associated with desirable outcomes for ARDS patients. Desirable outcomes refer to blood chemistry, heart rate, breathing cycle measurements, etc. being in acceptable ranges for that patient (see Section \ref{['sec:reward']}). A full dictionary of all variables is made available in Table \ref{['tab:variables']}. The methods by which actions are computed are visualized in Figure \ref{['fig:algorithms']}. The Pulse Physiology Engine is used to advance the simulation of the patient's health state (though learned approximations are used to estimate the patient's respiratory dynamics in some policies).
• Figure 2: An overview of the policies used to control ventilators in this work: a) Random Selection, b) Maximum Intervention, c) ARDSnet Protocol, d) Sampling-based Model Predictive Control, e) Embed To Control with Sampling-based Model Predictive Control, and f) Embed To Control with Model Predictive Path Integral Control.
• Figure 3: The total accumulated reward that each policy has earned at the end of the task, averaged over all patients ($N=100$). Shown are $95\%$ confidence intervals.