Bridging the Generalisation Gap: Synthetic Data Generation for Multi-Site Clinical Model Validation

TL;DR

The paper tackles the challenge of generalising clinical ML models across heterogeneous healthcare settings where real data are often inaccessible and distributional shifts are complex. It introduces a structured synthetic data generation framework with explicit control over site prevalence, hierarchical subgroup effects, and feature interactions to enable targeted benchmarking of robustness, fairness, and transportability. Through controlled experiments, it demonstrates faithful ground-truth preservation, accurate site-prevalence control, and insights into how model complexity interacts with site-specific effects to reveal generalisation failures absent in standard internal validation. The open-source toolkit provides a reproducible, interpretable platform for evaluating domain adaptation, fairness interventions, and federated-learning strategies in clinical AI.

Abstract

Ensuring the generalisability of clinical machine learning (ML) models across diverse healthcare settings remains a significant challenge due to variability in patient demographics, disease prevalence, and institutional practices. Existing model evaluation approaches often rely on real-world datasets, which are limited in availability, embed confounding biases, and lack the flexibility needed for systematic experimentation. Furthermore, while generative models aim for statistical realism, they often lack transparency and explicit control over factors driving distributional shifts. In this work, we propose a novel structured synthetic data framework designed for the controlled benchmarking of model robustness, fairness, and generalisability. Unlike approaches focused solely on mimicking observed data, our framework provides explicit control over the data generating process, including site-specific prevalence variations, hierarchical subgroup effects, and structured feature interactions. This enables targeted investigation into how models respond to specific distributional shifts and potential biases. Through controlled experiments, we demonstrate the framework's ability to isolate the impact of site variations, support fairness-aware audits, and reveal generalisation failures, particularly highlighting how model complexity interacts with site-specific effects. This work contributes a reproducible, interpretable, and configurable tool designed to advance the reliable deployment of ML in clinical settings.

Figures (4)

• Figure 1: Comparison of specified (true) and recovered (observed) feature effect sizes (logits) from logistic regression models trained on generated datasets of varying sizes (N=1k, 5k, 10k). Points represent individual features; alignment along the diagonal indicates accurate effect recovery.
• Figure 2: Relative recovery error of feature effect sizes as a function of the true effect magnitude for different dataset sizes (N). Error diminishes as both sample size and effect magnitude increase, indicating improved recovery precision.
• Figure 3: Observed vs. Target Outcome Prevalence per Site. Each point represents a simulated clinical site. Error bars show 95% bootstrapped confidence intervals for the observed prevalence within generated datasets of different sizes (N=1k, 5k, 10k). Observed prevalences closely match target values, with decreasing variability at larger sample sizes.
• Figure 4: Model Performance Degradation on External Sites with Increasing Site-Feature Interaction Effects. Performance degradation is the drop in AUROC between internal cross-validation and external site validation. Higher values indicate poorer generalisability. Models: Logistic Regression (LR), Random Forest (RF), Gradient Boosting (GB), XGBoost (XGB).