Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed

Minjae Kwon, Josephine Lamp, Lu Feng

TL;DR

The paper tackles the cross-cutting problem of safety generalization under distribution shift in safe reinforcement learning, using diabetes management as a safety-critical testbed. It demonstrates a notable safety generalization gap where training-time constraints fail on unseen patients, and proposes a runtime, algorithm-agnostic predictive shielding framework aided by Basis-Adaptive Neural ODEs (BA-NODE) to forecast patient-specific glucose trajectories and prune unsafe actions. A unified diabetes simulator and an OOD safety benchmark enable rigorous evaluation across diabetes types and age groups, showing that shielding yields time-in-range gains up to about 14% and reduces clinical risk across eight safe RL algorithms. The work provides a practical platform and methodological blueprint for deploying safe RL in safety-critical domains under distribution shift, with implications for clinical decision support and beyond.

Abstract

Safe Reinforcement Learning (RL) algorithms are typically evaluated under fixed training conditions. We investigate whether training-time safety guarantees transfer to deployment under distribution shift, using diabetes management as a safety-critical testbed. We benchmark safe RL algorithms on a unified clinical simulator and reveal a safety generalization gap: policies satisfying constraints during training frequently violate safety requirements on unseen patients. We demonstrate that test-time shielding, which filters unsafe actions using learned dynamics models, effectively restores safety across algorithms and patient populations. Across eight safe RL algorithms, three diabetes types, and three age groups, shielding achieves Time-in-Range gains of 13--14\% for strong baselines such as PPO-Lag and CPO while reducing clinical risk index and glucose variability. Our simulator and benchmark provide a platform for studying safety under distribution shift in safety-critical control domains. Code is available at https://github.com/safe-autonomy-lab/GlucoSim and https://github.com/safe-autonomy-lab/GlucoAlg.

Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed

TL;DR

The paper tackles the cross-cutting problem of safety generalization under distribution shift in safe reinforcement learning, using diabetes management as a safety-critical testbed. It demonstrates a notable safety generalization gap where training-time constraints fail on unseen patients, and proposes a runtime, algorithm-agnostic predictive shielding framework aided by Basis-Adaptive Neural ODEs (BA-NODE) to forecast patient-specific glucose trajectories and prune unsafe actions. A unified diabetes simulator and an OOD safety benchmark enable rigorous evaluation across diabetes types and age groups, showing that shielding yields time-in-range gains up to about 14% and reduces clinical risk across eight safe RL algorithms. The work provides a practical platform and methodological blueprint for deploying safe RL in safety-critical domains under distribution shift, with implications for clinical decision support and beyond.

Abstract

Safe Reinforcement Learning (RL) algorithms are typically evaluated under fixed training conditions. We investigate whether training-time safety guarantees transfer to deployment under distribution shift, using diabetes management as a safety-critical testbed. We benchmark safe RL algorithms on a unified clinical simulator and reveal a safety generalization gap: policies satisfying constraints during training frequently violate safety requirements on unseen patients. We demonstrate that test-time shielding, which filters unsafe actions using learned dynamics models, effectively restores safety across algorithms and patient populations. Across eight safe RL algorithms, three diabetes types, and three age groups, shielding achieves Time-in-Range gains of 13--14\% for strong baselines such as PPO-Lag and CPO while reducing clinical risk index and glucose variability. Our simulator and benchmark provide a platform for studying safety under distribution shift in safety-critical control domains. Code is available at https://github.com/safe-autonomy-lab/GlucoSim and https://github.com/safe-autonomy-lab/GlucoAlg.
Paper Structure (72 sections, 1 theorem, 66 equations, 15 figures, 16 tables, 1 algorithm)

This paper contains 72 sections, 1 theorem, 66 equations, 15 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Generalization and Distribution Shift.
  4. Runtime Safety Enforcement and Shielding.
  5. Unified Diabetes Simulator
  6. Physiological Model
  7. MDP Interface
  8. Personalized Dynamics Learning via Basis-Adaptive Neural ODEs
  9. Variate-Aware History Encoding
  10. Latent Dynamics via an ODE Ensemble
  11. Context-Conditioned Function-Space Combination
  12. Test-Time Predictive Shielding
  13. Shielding Architecture: Action Masking
  14. Shield Decision Logic
  15. Gating Mechanism.
  16. ...and 57 more sections

Key Result

Theorem 5.2

Suppose the shield uses a stricter pruning threshold $G_{\mathrm{shield}}^{\downarrow} = G_{\mathrm{fail}}^{\downarrow} + \varepsilon$. If the predictor is $(\varepsilon,\alpha)$-reliable, then any action $a$ permitted by the shield satisfies

Figures (15)

  • Figure 1: Predictive Verification Flow. The shield intercepts the policy's action distribution and identifies the top-$k$ candidate actions. It queries BA-NODE dynamics model to predict the blood glucose trajectory for each candidate. Actions resulting in predicted safety violations (hypo- or hyperglycemia) are masked. Additionally, immediate clinical overrides (e.g., rescue carbs) are applied alongside these predictive checks.
  • Figure 2: Trade-off between Risk Index (lower is better) and TIR (higher is better) across algorithms. Faint points show per-seed results; larger outlined markers show seed-averaged means.
  • Figure 3: Patient Parameters: Kernel density estimates of key virtual-patient parameters across child, adolescent, and adult cohorts. The distributions reflect cohort-level morphology and basal physiology, including body weight, basal glucose and insulin concentrations, endogenous glucose production, and glucose/insulin distribution volumes, highlighting age-dependent shifts in metabolic state.
  • Figure 4: Correlation heatmap (Pearson) among key clinical metrics, including time-in-range (TIR), glucose risk index, and coefficient of variation (CV). The heatmap illustrates how reward-aligned metrics co-vary with measures of glycemic risk and variability.
  • Figure 5: Relationship between episode reward and time-in-range (TIR). Each point represents one training seed; bold markers denote per-algorithm means. The positive correlation indicates that higher reward corresponds to improved TIR across algorithms and random seeds.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Definition 5.1: One-sided $(\varepsilon,\alpha)$-reliability
  • Theorem 5.2: Hypoglycemia Safety
  • proof