Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Addressing degeneracies in latent interpolation for diffusion models

Erik Landolsi, Fredrik Kahl

TL;DR

This work identifies and analyzes degeneracies that arise when interpolating latent latents from inverted diffusion model inputs as the number of sources grows. It shows that small, deterministic biases in latent statistics are amplified by norm-based interpolation, leading to degraded image quality even before degeneration is visible. The authors propose a mean-adjusted interpolation framework that separates a deterministic latent component from a stochastic one, interpolating the latter with norm-aware methods (notably a channel-wise mean adjustment) to mitigate degeneration. Across ImageNet experiments using Stable Diffusion variants, the channel-wise mean adjustment substantially improves FID and CLIP distances for larger $N$, enabling more reliable data augmentation and morphing with diffusion models.

Abstract

There is an increasing interest in using image-generating diffusion models for deep data augmentation and image morphing. In this context, it is useful to interpolate between latents produced by inverting a set of input images, in order to generate new images representing some mixture of the inputs. We observe that such interpolation can easily lead to degenerate results when the number of inputs is large. We analyze the cause of this effect theoretically and experimentally, and suggest a suitable remedy. The suggested approach is a relatively simple normalization scheme that is easy to use whenever interpolation between latents is needed. We measure image quality using FID and CLIP embedding distance and show experimentally that baseline interpolation methods lead to a drop in quality metrics long before the degeneration issue is clearly visible. In contrast, our method significantly reduces the degeneration effect and leads to improved quality metrics also in non-degenerate situations.

Addressing degeneracies in latent interpolation for diffusion models

TL;DR

This work identifies and analyzes degeneracies that arise when interpolating latent latents from inverted diffusion model inputs as the number of sources grows. It shows that small, deterministic biases in latent statistics are amplified by norm-based interpolation, leading to degraded image quality even before degeneration is visible. The authors propose a mean-adjusted interpolation framework that separates a deterministic latent component from a stochastic one, interpolating the latter with norm-aware methods (notably a channel-wise mean adjustment) to mitigate degeneration. Across ImageNet experiments using Stable Diffusion variants, the channel-wise mean adjustment substantially improves FID and CLIP distances for larger , enabling more reliable data augmentation and morphing with diffusion models.

Abstract

There is an increasing interest in using image-generating diffusion models for deep data augmentation and image morphing. In this context, it is useful to interpolate between latents produced by inverting a set of input images, in order to generate new images representing some mixture of the inputs. We observe that such interpolation can easily lead to degenerate results when the number of inputs is large. We analyze the cause of this effect theoretically and experimentally, and suggest a suitable remedy. The suggested approach is a relatively simple normalization scheme that is easy to use whenever interpolation between latents is needed. We measure image quality using FID and CLIP embedding distance and show experimentally that baseline interpolation methods lead to a drop in quality metrics long before the degeneration issue is clearly visible. In contrast, our method significantly reduces the degeneration effect and leads to improved quality metrics also in non-degenerate situations.
Paper Structure (18 sections, 9 equations, 12 figures)

This paper contains 18 sections, 9 equations, 12 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Latent space interpolation
  4. Latent diffusion models
  5. Interpolation notation
  6. Baseline interpolation options
  7. Degenerate interpolation output
  8. Initial investigation
  9. Bias amplification
  10. The origin of latent bias
  11. The effect of latent bias
  12. Alternative inversion procedures
  13. Mean-adjusted interpolation
  14. Experimental results
  15. Discussion
  16. ...and 3 more sections

Figures (12)

  • Figure 1: Images generated from centroids of $N$ latents obtained from $N$ input images of ImageNet class "tree frog" for $N = 2, 8, 32, 48, 64, 96$. Top row: Linear interpolation. Middle row: Linear interpolation with fixed normalization. Bottom row: channel-wise mean adjustment (our suggested method).
  • Figure 2: Interpolation paths produced by the methods from Section \ref{['sec:baseline_interp_options']} for a 2D toy example. The gray circle represents latents with norm $\sqrt{L}$, which is where randomly sampled latents are typically located.
  • Figure 3: Images generated from centroids created using the recipe in Section \ref{['sec:initial_investigation']}. ImageNet class "tree frog", $N = 2, 8, 32, 48, 64, 96$, fixed normalization.
  • Figure 4: Channel-wise mean of the first two channels of $f_\textrm{FIX}$ centroids computed from $N$ inverted input images, plotted for 8 selected ImageNet classes. Left: SD 1.5, right: SD 3.5. The x-axes have a square-root scale to illustrate the linear dependence on $\sqrt{N}$.
  • Figure 5: Images generated from centroids created using the recipe from Section \ref{['sec:initial_investigation']}, to ensure i.i.d. examples, using SD 3.5 to avoid the non-zero terminal SNR issue. $N = 2, 8, 32, 48, 64, 96$ (increasing to the right), fixed normalization.
  • ...and 7 more figures