Report on the AAPM Grand Challenge on deep generative modeling for learning medical image statistics

TL;DR

This study presents a two-stage, domain-focused evaluation framework for learning medical image statistics using deep generative models (DGMs). By leveraging the VICTRE-derived breast phantom SOMs and a comprehensive radiomic-statistic metric, the challenge demonstrates that high perceptual quality (FID performance) does not guarantee accurate replication of clinically meaningful image statistics, underscoring the need for domain-specific assessments and artifact-aware model development. The results reveal diverse DGM approaches (notably conditional latent diffusion models) and highlight artifacts common across methods, emphasizing the importance of evaluating stability, diversity, and per-class fidelity for reliable in silico trials. Overall, the work advances how to benchmark DGMs for medical imaging applications, informing safer, more effective deployment and guiding future improvements in DGM design and evaluation.

Abstract

The findings of the 2023 AAPM Grand Challenge on Deep Generative Modeling for Learning Medical Image Statistics are reported in this Special Report. The goal of this challenge was to promote the development of deep generative models (DGMs) for medical imaging and to emphasize the need for their domain-relevant assessment via the analysis of relevant image statistics. As part of this Grand Challenge, a training dataset was developed based on 3D anthropomorphic breast phantoms from the VICTRE virtual imaging toolbox. A two-stage evaluation procedure consisting of a preliminary check for memorization and image quality (based on the Frechet Inception distance (FID)), and a second stage evaluating the reproducibility of image statistics corresponding to domain-relevant radiomic features was developed. A summary measure was employed to rank the submissions. Additional analyses of submissions was performed to assess DGM performance specific to individual feature families, and to identify various artifacts. 58 submissions from 12 unique users were received for this Challenge. The top-ranked submission employed a conditional latent diffusion model, whereas the joint runners-up employed a generative adversarial network, followed by another network for image superresolution. We observed that the overall ranking of the top 9 submissions according to our evaluation method (i) did not match the FID-based ranking, and (ii) differed with respect to individual feature families. Another important finding from our additional analyses was that different DGMs demonstrated similar kinds of artifacts. This Grand Challenge highlighted the need for domain-specific evaluation to further DGM design as well as deployment. It also demonstrated that the specification of a DGM may differ depending on its intended use.

Figures (15)

• Figure 1: The DGM challenge workflow.
• Figure 2: Sample images from the training dataset corresponding to four classes: dense (upper left), heterogeneous (upper right), scattered (lower left), and fatty (lower right). Class information was not provided explicitly to the participants.
• Figure 3: Images generated by the top three approaches alongside the images from the training data.
• Figure 4: FID and memorization metric scores for the submissions alongside the FID and memorization metric scores of a baseline StyleGAN2 model trained in-house
• Figure 5: An image each from three submissions that were ruled out in the first stage of evaluation. The images in the left and center positions correspond to the submissions that did not pass the FID threshold, whereas the rightmost image corresponds to the submission that did not pass both the FID and the memorization thresholds.