Distribution-Free Uncertainty Quantification in Mechanical Ventilation Treatment: A Conformal Deep Q-Learning Framework

TL;DR

This work tackles uncertainty and distribution shift in reinforcement learning for mechanical ventilation by introducing ConformalDQN, a distribution-free, conformal prediction–augmented DDQN. The framework integrates a state-conditioned action-probability estimator and a conformal predictor to yield uncertainty-aware action selection, calibrated to maintain prescribed coverage. Evaluated on the MIMIC-IV dataset, ConformalDQN achieves higher estimated 90-day survival and produces action distributions aligned with clinical lung-protective guidelines, while maintaining stable Q-values in out-of-distribution scenarios. The approach provides interpretable confidence in decisions, enabling safer human-in-the-loop adoption and potential extension to other safety-critical healthcare and non-healthcare applications. Future work includes extending to continuous action spaces and developing state-conditioned conformal calibration to further enhance performance under patient heterogeneity.

Abstract

Mechanical Ventilation (MV) is a critical life-support intervention in intensive care units (ICUs). However, optimal ventilator settings are challenging to determine because of the complexity of balancing patient-specific physiological needs with the risks of adverse outcomes that impact morbidity, mortality, and healthcare costs. This study introduces ConformalDQN, a novel distribution-free conformal deep Q-learning approach for optimizing mechanical ventilation in intensive care units. By integrating conformal prediction with deep reinforcement learning, our method provides reliable uncertainty quantification, addressing the challenges of Q-value overestimation and out-of-distribution actions in offline settings. We trained and evaluated our model using ICU patient records from the MIMIC-IV database. ConformalDQN extends the Double DQN architecture with a conformal predictor and employs a composite loss function that balances Q-learning with well-calibrated probability estimation. This enables uncertainty-aware action selection, allowing the model to avoid potentially harmful actions in unfamiliar states and handle distribution shifts by being more conservative in out-of-distribution scenarios. Evaluation against baseline models, including physician policies, policy constraint methods, and behavior cloning, demonstrates that ConformalDQN consistently makes recommendations within clinically safe and relevant ranges, outperforming other methods by increasing the 90-day survival rate. Notably, our approach provides an interpretable measure of confidence in its decisions, which is crucial for clinical adoption and potential human-in-the-loop implementations.

Figures (4)

• Figure 1: Interaction of RL agent with environment (patient).
• Figure 2: Overview of the Method. (1)Preparing a structured dataset of past experiences.(2) Training a Double Deep Q-Network (DDQN) model and an action probability estimator (3) Calibration: utilizing the calibration set to determine a conformity threshold.(4) Action selection process: the state is fed into both the DDQN model and the action probability estimator, where actions are selected based on Q-values filtered through the conformity threshold to filter reliable actions. (5) Evaluation: the model's performance is evaluated using the test data.
• Figure 3: Distribution of ventilator settings chosen by different policies. PEEP: Positive End-Expiratory Pressure (cmH2O); FiO2: Fraction of Inspired Oxygen (%); Adjusted Tidal Volume (ml/kg).
• Figure 4: Comparison of mean initial Q-values for in-distribution (ID) and out-of-distribution (OOD) scenarios. Error bars represent variances. The horizontal line indicates the maximum expected return.