Enabling Granular Subgroup Level Model Evaluations by Generating Synthetic Medical Time Series

TL;DR

This work introduces Enhanced TimeAutoDiff, a latent-diffusion framework augmented with distribution-alignment losses to generate synthetic ICU time-series data that serve as reliable proxies for real data both in global model evaluation and fine-grained subgroup analysis. By combining Maximum Mean Discrepancy and consistency regularization, the authors substantially reduce the TRTS evaluation gap (to $\Delta_{TRTS} \leq 0.014$ AUROC) while preserving training utility ($\Delta_{TSTR} \approx 0.01$). The method enables large, conditionally sampled cohorts that improve subgroup-level AUROC estimation accuracy for 32 intersectional groups, outperforming small real test sets in 72–84% of cases and reducing subgroup errors by up to 50%. These results offer a practical, privacy-preserving blueprint for trustworthy, granular medical AI evaluation, with open-source code and evaluation pipelines to facilitate adoption across institutions and regulators.

Abstract

We present a novel framework for leveraging synthetic ICU time-series data not only to train but also to rigorously and trustworthily evaluate predictive models, both at the population level and within fine-grained demographic subgroups. Building on prior diffusion and VAE-based generators (TimeDiff, HealthGen, TimeAutoDiff), we introduce \textit{Enhanced TimeAutoDiff}, which augments the latent diffusion objective with distribution-alignment penalties. We extensively benchmark all models on MIMIC-III and eICU, on 24-hour mortality and binary length-of-stay tasks. Our results show that Enhanced TimeAutoDiff reduces the gap between real-on-synthetic and real-on-real evaluation (``TRTS gap'') by over 70\%, achieving $Δ_{TRTS} \leq 0.014$ AUROC, while preserving training utility ($Δ_{TSTR} \approx 0.01$). Crucially, for 32 intersectional subgroups, large synthetic cohorts cut subgroup-level AUROC estimation error by up to 50\% relative to small real test sets, and outperform them in 72--84\% of subgroups. This work provides a practical, privacy-preserving roadmap for trustworthy, granular model evaluation in critical care, enabling robust and reliable performance analysis across diverse patient populations without exposing sensitive EHR data, contributing to the overall trustworthiness of Medical AI.

Figures (7)

• Figure 1: Workflow for Generative Model Training and Evaluation of Synthetic ICU Data. The real ICU datasets (MIMIC and eICU) are split into training ($R^{train}$, 45%), holdout ($R^{holdout}$, 45%), and validation ($R^{holdout-val}$, 10%) subsets. A generative model is trained on $R^{train}$ to produce synthetic data $S$. The synthetic data are assessed through two complementary evaluations: fidelity (green- intrinsic comparison with real data) and utility (purple- extrinsic evaluation via downstream classifier performance). Solid lines indicate training flows, while dashed lines denote evaluation processes.
• Figure 2: Training Utility workflow.
• Figure 3: Evaluation Utility workflow.
• Figure 4: Subgroup Utility workflow.
• Figure 5: Model Comparison - Discriminative Fidelity This figure displays bar charts comparing the Discriminative Score of HealthGen, TimeAutoDiff, Enhanced TimeAutoDiff and TimeDiff across different tasks and datasets. The lower the scores, the better.