A Framework for Evaluating Predictive Models Using Synthetic Image Covariates and Longitudinal Data

TL;DR

A novel framework to synthesize data sets with complex covariates, such as medical images, linked to simulated longitudinal patient outcomes, and successfully applies the methodology to OCT images is presented.

Abstract

We present a novel framework for synthesizing patient data with complex covariates (e.g., eye scans) paired with longitudinal observations (e.g., visual acuity over time), addressing privacy concerns in healthcare research. Our approach introduces controlled association in latent spaces generating each data modality, enabling the creation of complex covariate-longitudinal observation pairs. This framework facilitates the development of predictive models and provides openly available benchmarking datasets for healthcare research. We demonstrate our framework using optical coherence tomography (OCT) scans, though it is applicable across domains. Using 109,309 2D OCT scan slices, we trained an image generative model combining a variational autoencoder and a diffusion model. Longitudinal observations were simulated using a nonlinear mixed effect (NLME) model from a low-dimensional space of random effects. We generated 1.1M OCT scan slices paired with five sets of longitudinal observations at controlled association levels (100%, 50%, 10%, 5.26%, and 2% of between-subject variability). To assess the framework, we modeled synthetic longitudinal observations with another NLME model, computed empirical Bayes estimates of random effects, and trained a ResNet to predict these estimates from synthetic OCT scans. We then incorporated ResNet predictions into the NLME model for patient-individualized predictions. Prediction accuracy on withheld data declined as intended with reduced association between images and longitudinal measurements. Notably, in all but the 2% case, we achieved within 50% of the theoretical best possible prediction on withheld data, demonstrating our ability to detect even weak signals. This confirms the effectiveness of our framework in generating synthetic data with controlled levels of association, providing a valuable tool for healthcare research.

Figures (5)

• Figure 1: The diagram illustrates the flow from the latent space vector $\boldsymbol{z}$, which is used both to generate medical images via the SD model and to derive a subset vector $\boldsymbol{\eta} \subseteq \boldsymbol{z}$. Additional noise is applied to $\boldsymbol{\eta}$, resulting in $\hat{\boldsymbol{\eta}}$, which is then employed in the NLME model to produce synthetic longitudinal data. This process allows for controlled tuning of the correlation between the synthetic images and the generated longitudinal data outcomes.
• Figure 2: Training procedure leveraging the NLME model. Starting with synthetic longitudinal data, the NLME model approximates the random effects, yielding $\boldsymbol{\eta_\text{approx}}$. This approximation aids in training the neural network to relate images to their corresponding random effects.
• Figure 3: Procedure for predicting longitudinal data using the trained model. An image, when introduced to the trained neural network, predicts the associated random effects, denoted as $\boldsymbol{\eta}_{\text{pred}}$. These effects, when incorporated into the NLME model, yield the corresponding longitudinal data. The predicted longitudinal data can then be compared with the original synthetic longitudinal data for validation.
• Figure 4: Comparison of original OCT images from the validation set (left) and their respective reconstructions (right) derived from the SD model when it was conditioned on the latent space vector $\boldsymbol{z}$ created by the VAE encoder.
• Figure 5: Generated OCT images delineating the effects of selected latent variable dimensions. Rows, in descending order, depict modifications in vertical positioning, tilt, luminance, and curvature. The central column, with the selected latent variables equal to zero, serves as a reference. Horizontal deviations from the central column signify alterations in a singular latent variable dimension, with all other dimensions held constant.