Beyond the Generative Learning Trilemma: Generative Model Assessment in Data Scarcity Domains

TL;DR

This study tackles data scarcity by extending the Generative Learning Trilemma to include utility, robustness, and privacy, and by evaluating VAEs, GANs, and Diffusion Models (DMs) across four real-world datasets in medicine and precision agriculture. A robust framework of six metrics (fidelity, diversity, sampling speed, utility, robustness, privacy) assesses synthetic data quality and downstream impact, with class-conditional CVAEs, StyleGAN2, and Latent Diffusion Models implemented and tested under limited data. Results show DMs deliver the best fidelity and diversity but suffer from slower sampling, GANs balance fidelity and privacy with reasonable speed, and VAEs excel in sampling speed albeit with lower fidelity/diversity; combining synthetic data with geometric augmentation often yields the largest gains and improved robustness. The findings provide practical guidance for selecting DGMs in data-scarce domains and highlight the need for hybrid models and broader privacy/attack evaluations to enhance real-world applicability.

Abstract

Data scarcity remains a critical bottleneck impeding technological advancements across various domains, including but not limited to medicine and precision agriculture. To address this challenge, we explore the potential of Deep Generative Models (DGMs) in producing synthetic data that satisfies the Generative Learning Trilemma: fidelity, diversity, and sampling efficiency. However, recognizing that these criteria alone are insufficient for practical applications, we extend the trilemma to include utility, robustness, and privacy, factors crucial for ensuring the applicability of DGMs in real-world scenarios. Evaluating these metrics becomes particularly challenging in data-scarce environments, as DGMs traditionally rely on large datasets to perform optimally. This limitation is especially pronounced in domains like medicine and precision agriculture, where ensuring acceptable model performance under data constraints is vital. To address these challenges, we assess the Generative Learning Trilemma in data-scarcity settings using state-of-the-art evaluation metrics, comparing three prominent DGMs: Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Diffusion Models (DMs). Furthermore, we propose a comprehensive framework to assess utility, robustness, and privacy in synthetic data generated by DGMs. Our findings demonstrate varying strengths among DGMs, with each model exhibiting unique advantages based on the application context. This study broadens the scope of the Generative Learning Trilemma, aligning it with real-world demands and providing actionable guidance for selecting DGMs tailored to specific applications.

Figures (7)

• Figure 1: Conceptual overview illustrating the intuition behind the six metrics defined for evaluating DGMs: fidelity, diversity, sampling speed, utility, robustness, and privacy.
• Figure 2: Overview of the methodology for the classification downstream task used to assess utility and robustness. Two baseline approaches are considered: one using real data and another incorporating Geometric DA for classifier training. Data Anonymization is then performed using only synthetic data generated by three deep generative models (DGMs): Variational Autoencoder (VAE), Generative Adversarial Network (GAN), and Diffusion Model (DM). In this phase, sampling speed ($\mathcal{S}$), fidelity ($\mathcal{F}$), diversity ($\mathcal{D}$), and privacy ($\mathcal{P}$) are also evaluated. Finally, Synthetic DA combines real data with synthetic samples from the DGMs, while Combined DA further integrates Geometric DA.
• Figure 3: Architectures of Deep Generative Models. (Top) VAE comprising an encoder that maps input data $s$ to a latent variable $z$, and a decoder reconstructing the output $s'$. (Middle) GAN including a generator, which maps latent variable $z$ to synthetic data $s$, and a discriminator distinguishing real data $r$ from generated samples. (Bottom) DM illustrating the forward process that adds noise to data $s$ iteratively to generate $s_1, s_2, \dots$, and a backward process that learns to reverse this chain, obtaining latent representation $z$.
• Figure 4: Comparison of synthetic images generated by DGMs with real images. For each of the four datasets used in the analysis, two images are presented.
• Figure 5: This figure compares the utility of synthetic data generated by the three DGMs (VAE, GAN, and DM) across the four datasets, using the accuracy of the classification task. It examines the effect of increasing the quantity of synthetic data by doubling and tripling the size of the training set. Additionally, two baselines, real data and Geometric DA, are included for reference.