Investigation into using stochastic embedding representations for evaluating the trustworthiness of the Fréchet Inception Distance

TL;DR

The paper tackles the reliability of the Fréchet Inception Distance (FID) for assessing synthetic medical images, where standard natural-image embeddings may be misleading. It adopts Monte Carlo dropout to quantify uncertainty in both embedding representations and the FID, introducing metrics such as $pVar$ and $\mathrm{vFID}$ to capture predictive variance across stochastic evaluations. Empirical results show that the uncertainty proxy $\sigma_{\mathrm{FID}}$ tends to increase with the degree of out-of-distribution (OOD) in test data, suggesting its potential as a usable indicator of FID trustworthiness, while $pVar$ exhibits less consistent behavior. The work provides a framework for uncertainty-aware evaluation of generative models in high-stakes settings like healthcare, highlighting when and how FID-based assessments may be trusted and where they may require caution.

Abstract

Feature embeddings acquired from pretrained models are widely used in medical applications of deep learning to assess the characteristics of datasets; e.g. to determine the quality of synthetic, generated medical images. The Fréchet Inception Distance (FID) is one popular synthetic image quality metric that relies on the assumption that the characteristic features of the data can be detected and encoded by an InceptionV3 model pretrained on ImageNet1K (natural images). While it is widely known that this makes it less effective for applications involving medical images, the extent to which the metric fails to capture meaningful differences in image characteristics is not obviously known. Here, we use Monte Carlo dropout to compute the predictive variance in the FID as well as a supplemental estimate of the predictive variance in the feature embedding model's latent representations. We show that the magnitudes of the predictive variances considered exhibit varying degrees of correlation with the extent to which test inputs (ImageNet1K validation set augmented at various strengths, and other external datasets) are out-of-distribution relative to its training data, providing some insight into the effectiveness of their use as indicators of the trustworthiness of the FID.

Figures (4)

• Figure 1: Example inputs from left to right: an ImageNet1K validation set image, the same image augmented with additive random Gaussian noise with a standard deviation defined by 1 $\%$ of the maximum amplitude, augmented with four random miniaturised CelebA images randomly rotated and overlaid, augmented with four random miniaturised mammography images randomly rotated and overlaid, a random CelebA image, and a mammogram.
• Figure 2: a) FID and $\sigma$FID vs. augmentation strength and b) pVar vs augmentation strength for the equal augmentation experiment.
• Figure 3: Plots of a) Mean FID over MCD samples, and (inverted vertical axis) top-5 accuracy on ImageNet1K classification, b) MAE and (inverted vertical axis) MS-SSIM, c) pVar, and d) $\sigma \text{FID}$ vs noise augmentation (additive random Gaussian noise with a standard deviation defined by some percentage of the image's maximum amplitude, referred to here as strength). e) Decomposition of vFID vs augmentation strength where $a = \|\mu^{j}_{\hat{x}}-\mu_y\|_2^2$, $b = \text{tr}(\Sigma^{j}_{\hat{x}}+\Sigma_y)$ and $c = \text{tr}(\Sigma^{j}_{\hat{x}} \Sigma_y)^{\frac{1}{2}}$, so $\text{FID} = a + b - 2c$. f) First term of the FID computed using the mean over the MCD samples of the test latent means (blue) and expected standard deviation of the latent means across MCD samples (red) vs augmentation strength; the latter correlates with c) as expected. g) Norm of the test embeddings vs augmentation strength, h) examples of noise augmented images.
• Figure 4: Results of the same experiment as Fig. \ref{['fig:sens_inputs']} but with a fixed test set (MCD applied to unaugmented inputs, keeping pVar constant for the plot) and increasingly augmented reference set (single MCD sample). Here, we do not see a drop in $\sigma$FID in plot c, which suggests the discrepancy in image fidelity could be the reason for the increase observed in Fig. \ref{['fig:sens_inputs']}d, and that the decrease at higher strength augmentations is due to the decrease in pVar.