On Safer Reinforcement Learning Policies for Sedation and Analgesia in Intensive Care

TL;DR

The study treats ICU sedation/analgesia as an offline, partially observable RL problem and investigates how reward design affects safety. Using 47,144 ICU stays from MIMIC-IV, it trains four-continuous-action policies with a GRU-based state representation and TD3_BC-like offline learning, comparing a pain-only objective to a pain-plus-mortality objective. The results show that prioritizing pain relief alone can be associated with higher 30-day mortality, whereas including mortality in the objective yields policies linked to lower mortality and pain, illustrating the importance of long-term outcomes for safety. The work expands to a richer action space and demonstrates the challenges of evaluation under imperfect information and observational data, signaling the need for causal validation and multi-center studies to establish clinical impact.

Abstract

Pain management in intensive care usually involves complex trade-offs between therapeutic goals and patient safety, since both inadequate and excessive treatment may induce serious sequelae. Reinforcement learning can help address this challenge by learning medication dosing policies from retrospective data. However, prior work on sedation and analgesia has optimized for objectives that do not value patient survival while relying on algorithms unsuitable for imperfect information settings. We investigated the risks of these design choices by implementing a deep reinforcement learning framework to suggest hourly medication doses under partial observability. Using data from 47,144 ICU stays in the MIMIC-IV database, we trained policies to prescribe opioids, propofol, benzodiazepines, and dexmedetomidine according to two goals: reduce pain or jointly reduce pain and mortality. We found that, although the two policies were associated with lower pain, actions from the first policy were positively correlated with mortality, while those proposed by the second policy were negatively correlated. This suggests that valuing long-term outcomes could be critical for safer treatment policies, even if a short-term goal remains the primary objective.

Figures (10)

• Figure 1: Schematic of our data-driven reinforcement learning (RL) framework for sedation and analgesia in the intensive care unit (ICU). We used offline RL with partial observability to train, validate, and test recurrent agents on 47,144 ICU stays. Specifically, we trained policies to recommend propofol, opioids, benzodiazepines, and dexmedetomidine doses per hour, seeking to minimize pain and mortality. The agents operated based on observations, including vital signs, pain reports, and laboratory values. We studied their behavior to elucidate which objectives encouraged safer strategies.
• Figure 2: Our dataset included both quantitative and ordinal observations, apart from four continuous medication actions per hour.
• Figure 3: Our actor-critic architecture addresses both imperfect information and offline learning. The recurrent representation network $\text{GRU}_{\psi}$ learns to capture the influence of past observations, thereby approximating the patient state. Subsequently, the behavior-regularized actor $\pi_{\phi}$ proposes actions based on those states, which are evaluated by the critics $Q_{\theta_1}$ and $Q_{\theta_2}$.
• Figure 4: Clinician-agent agreement for policies A ($\pi_{\phi_A}$) and B ($\pi_{\phi_B}$) in the test set. Bars show the mean similarity, overall and medication-specific, including opioids, propofol, benzodiazepines (BZD), and dexmedetomidine (DEX). Error bars indicate the standard deviation across patient stays.
• Figure 5: Correlation between clinician-agent agreement and mortality (left) or cumulative pain (right) for policy A ($\pi_{\phi_A}$) and policy B ($\pi_{\phi_B}$). Points indicate correlation estimates, and horizontal bars represent confidence intervals. The vertical dashed line denotes no association ($\rho$ = 0).