Trustworthy image-to-image translation: evaluating uncertainty calibration in unpaired training scenarios

TL;DR

This work tackles trustworthy image-to-image translation in unpaired settings for medical imaging by comparing cycleGAN and diffusion-based SynDiff and applying uncertainty quantification through Monte Carlo Dropout and deep ensembles. It introduces a data-efficient calibration scheme that uses augmented test sets to relate accuracy metrics like FID to predictive uncertainty (mPSD), enabling assessment of whether the estimated uncertainty meaningfully reflects model doubt. Across mammography and non-medical tasks, the study shows that (i) both models can reduce style-transfer FID while preserving tissue structure (CWSSIM) and (ii) uncertainty estimates correlate with image quality and can highlight hallucinations, supporting cautious deployment in data-scarce clinical domains. The approach provides a practical path toward trustworthy unpaired translation when ground truths are unavailable, though calibration is demonstrated on augmented subsets and requires broader validation and hyperparameter exploration for robust generalization.

Abstract

Mammographic screening is an effective method for detecting breast cancer, facilitating early diagnosis. However, the current need to manually inspect images places a heavy burden on healthcare systems, spurring a desire for automated diagnostic protocols. Techniques based on deep neural networks have been shown effective in some studies, but their tendency to overfit leaves considerable risk for poor generalisation and misdiagnosis, preventing their widespread adoption in clinical settings. Data augmentation schemes based on unpaired neural style transfer models have been proposed that improve generalisability by diversifying the representations of training image features in the absence of paired training data (images of the same tissue in either image style). But these models are similarly prone to various pathologies, and evaluating their performance is challenging without ground truths/large datasets (as is often the case in medical imaging). Here, we consider two frameworks/architectures: a GAN-based cycleGAN, and the more recently developed diffusion-based SynDiff. We evaluate their performance when trained on image patches parsed from three open access mammography datasets and one non-medical image dataset. We consider the use of uncertainty quantification to assess model trustworthiness, and propose a scheme to evaluate calibration quality in unpaired training scenarios. This ultimately helps facilitate the trustworthy use of image-to-image translation models in domains where ground truths are not typically available.

Figures (6)

• Figure 1: a) Plot of FID vs added noise for shoes $\leftrightarrow$ sketches (cycleGAN), showing that the FID increases with added noise for augmented images. This coarsely validates its use as an accuracy metric. Unaugmented shoe images produce poor quality outputs likely due to the model's handling of empty (low variability in pixel amplitude) regions of the input. The annotation provides an example output sketch produced from an unaugmented shoe input. Gaussian noise is added to each input image, with a mean of zero and a variance defined by the % of the amplitude of the max pixel value in the image (horizontal axis). b) Plot of FID vs mPSD for the same tasks. We see a relationship between FID and mPSD for augmented images.
• Figure 2: An augmented input shoe image, with its style-transferred sketch counterpart. The output does not feature the hallucinations seen in the output (annotation of Fig\ref{['fig:fidvsuncert_shoes']}) produced using the unaugmented version of the shoe image, despite using the same model. Adding noise removes empty (low variability in pixel amplitude) regions of the input prone to hallucinations.
• Figure 3: An example from each model for the VDM $\leftrightarrow$ CBIS task.
• Figure 4: a) An unadapted CBIS image patch, where the red box indicates a sparser region of tissue. b) The style-transferred (cycleGAN) counterpart to a), that contains hallucinated features in this sparse region. c) the pixel-wise predictive standard deviation ($\sigma$). The hallucinated region has correspondingly high $\sigma$. All images are shown in viridus colour scale to improve visualisation.
• Figure 5: a) Plot of FID (computed from single evaluation) vs added noise for VDM $\leftrightarrow$ CBIS (cycleGAN), showing that the FID correlates with added noise. This coarsely validates its use as an accuracy metric. Gaussian noise is added to each input image, with a mean of zero and a variance defined by the % of the amplitude of the max pixel value in the image (horizontal axis). b) Plot of the same FID vs mPSD (Monte Carlo Dropout) for VDM $\leftrightarrow$ CBIS.