Making Reconstruction FID Predictive of Diffusion Generation FID

TL;DR

Interpolated FID (iFID), a simple variant of rFID that exhibits a strong correlation with diffusion gFID, is proposed, and is the first metric to demonstrate a strong correlation with diffusion gFID.

Abstract

It is well known that the reconstruction FID (rFID) of a VAE is poorly correlated with the generation FID (gFID) of a latent diffusion model. We propose interpolated FID (iFID), a simple variant of rFID that exhibits a strong correlation with gFID. Specifically, for each element in the dataset, we retrieve its nearest neighbor (NN) in the latent space and interpolate their latent representations. We then decode the interpolated latent and compute the FID between the decoded samples and the original dataset. Additionally, we refine the claim that rFID correlates poorly with gFID, by showing that rFID correlates with sample quality in the diffusion refinement phase, whereas iFID correlates with sample quality in the diffusion navigation phase. Furthermore, we provide an explanation for why iFID correlates well with gFID, and why reconstruction metrics are negatively correlated with gFID, by connecting to results in the diffusion generalization and hallucination. Empirically, iFID is the first metric to demonstrate a strong correlation with diffusion gFID, achieving Pearson linear and Spearman rank correlations approximately 0.85. The source code is provided in https://github.com/tongdaxu/Making-rFID-Predictive-of-Diffusion-gFID.

Figures (7)

• Figure 1: Left two plots: The rFID values of VAEs are uncorrelated, or even negatively correlated with, the gFID values of diffusion models. Right two plots: iFID metric exhibits a strong positive correlation with the gFID values of diffusion models.
• Figure 2: The refinement and navigation phases are key components of the sampling process for SiT-XL trained with SD-VAE. In the refinement phase (small $t$), the sample generated from the noisy source is nearly identical to the source. In contrast, during the navigation phase (large $t$), the sample from the noisy source differs significantly from the source.
• Figure 3: Toy example illustrating how different properties of the latent space lead to different diffusion sampling results. Left two plots: The latent is an isolated 25 Gaussian mixture in a two-dimensional square grid and exhibits poor iFID, since the interpolated $\hat{z}$ does not lie on the data manifold. In this case, diffusion samples interpolating between nearby modes also fall outside the data manifold, leading to significant hallucination. Right two plots: The latent is a connected 25 Gaussian mixture and achieves good iFID, as the interpolated $\hat{z}$ remains on the data manifold. Consequently, diffusion samples interpolating between nearby modes stay within the data manifold, and hallucination is reduced.
• Figure 4: Toy example illustrating the difference between reconstruction-oriented latent and diffusion-oriented latent. Left two plots: The latent is an isolated $2$-mode Gaussian mixture and exhibits poor interoperability, since the interpolated $\hat{z}$ does not lie on the data manifold. In this case, diffusion samples interpolated between nearby modes also fall outside the data manifold, leading to hallucinations. Right two plots: The latent is an overlapping $2$-mode Gaussian mixture with good interoperability, as the interpolated $\hat{z}$ remains on the data manifold. Consequently, diffusion samples interpolated between nearby modes stay within the data manifold, and hallucinations are reduced.
• Figure 5: The visualization of decoded nearest neighbour latent NN($z$) and the interpolated latent $\hat{z}$. For reconstruction oriented VAEs, the NN($z$) is semantically different from $z$, and the interpolated $\hat{z}$ are invalid images. While for diffusion oriented VAEs, the NN($z$) is semantically similar to $z$, and the interpolated $\hat{z}$ are realistic images.