Metrics that matter: Evaluating image quality metrics for medical image generation

TL;DR

This study systematically benches no-reference image quality metrics (NRIQMs) for synthetic medical images against downstream segmentation utility, using brain MRI data (BRaTS tumours and IXI vascular images). By applying controlled perturbations (noise, boundary blur, internal gradients, and distribution shifts) and evaluating outputs from VAE, GAN, and DDPM architectures, the authors demonstrate that common NRQIMs often misrepresent model quality, are insensitive to localized anatomical details, and can diverge from downstream task performance. They show DDPMs frequently yield better downstream segmentation results than GANs or VAEs, even when upstream metrics imply large distributional gaps, highlighting the risk of relying solely on NRQIMs for clinical readiness. The work advocates a multifaceted validation framework that couples careful downstream task evaluation with selective upstream metrics to ensure the safety and clinical applicability of synthetic medical images. The authors provide actionable recommendations and emphasize the need for anatomy-aware metrics validated in medical contexts to support responsible translation of generative models into healthcare.

Abstract

Evaluating generative models for synthetic medical imaging is crucial yet challenging, especially given the high standards of fidelity, anatomical accuracy, and safety required for clinical applications. Standard evaluation of generated images often relies on no-reference image quality metrics when ground truth images are unavailable, but their reliability in this complex domain is not well established. This study comprehensively assesses commonly used no-reference image quality metrics using brain MRI data, including tumour and vascular images, providing a representative exemplar for the field. We systematically evaluate metric sensitivity to a range of challenges, including noise, distribution shifts, and, critically, localised morphological alterations designed to mimic clinically relevant inaccuracies. We then compare these metric scores against model performance on a relevant downstream segmentation task, analysing results across both controlled image perturbations and outputs from different generative model architectures. Our findings reveal significant limitations: many widely-used no-reference image quality metrics correlate poorly with downstream task suitability and exhibit a profound insensitivity to localised anatomical details crucial for clinical validity. Furthermore, these metrics can yield misleading scores regarding distribution shifts, e.g. data memorisation. This reveals the risk of misjudging model readiness, potentially leading to the deployment of flawed tools that could compromise patient safety. We conclude that ensuring generative models are truly fit for clinical purpose requires a multifaceted validation framework, integrating performance on relevant downstream tasks with the cautious interpretation of carefully selected no-reference image quality metrics.

Figures (8)

• Figure 1: A graph of different types of metrics used in generative medical image evaluation, categorised by modality.
• Figure 2: Overview of our methodology.
• Figure 3: Overview of the controlled perturbation experiments in Phase 1. We apply noise, morphological, and domain-shift manipulations to real medical images to evaluate the sensitivity of NRIQMs.
• Figure 4: Heatmap illustrating the sensitivity of each evaluation metric to various forms of noise perturbation.
• Figure 5: Heatmap illustrating the sensitivity of upstream evaluation metrics to morphological perturbations. Note the widespread lack of response across most metrics, contrasting with the downstream sensitivity shown in Table \ref{['tab:evany_morph']}.