Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusion

Nicholas Karris, Luke Durell, Javier Flores, Tegan Emerson

TL;DR

The paper reframes CLIP prompt embeddings as point clouds in Wasserstein space to study their geometry within Stable Diffusion. By exploiting permutation-invariance of cross-attention, embeddings are interpolated along optimal-transport geodesics, producing smoother intermediate images than standard linear or random couplings. The approach is validated on a large set of prompt pairs from Crisscrossed Captions, using LPIPS-based Perceptual Path Length to quantify path smoothness, with OT dominating CLIP and random methods, especially for more similar prompts. The work highlights a geometry-aware view of embedding spaces that improves image interpolation and suggests directions for richer metrics and broader applicability to newer diffusion models and multi-modal token spaces.

Abstract

It can be shown that Stable Diffusion has a permutation-invariance property with respect to the rows of Contrastive Language-Image Pretraining (CLIP) embedding matrices. This inspired the novel observation that these embeddings can naturally be interpreted as point clouds in a Wasserstein space rather than as matrices in a Euclidean space. This perspective opens up new possibilities for understanding the geometry of embedding space. For example, when interpolating between embeddings of two distinct prompts, we propose reframing the interpolation problem as an optimal transport problem. By solving this optimal transport problem, we compute a shortest path (or geodesic) between embeddings that captures a more natural and geometrically smooth transition through the embedding space. This results in smoother and more coherent intermediate (interpolated) images when rendered by the Stable Diffusion generative model. We conduct experiments to investigate this effect, comparing the quality of interpolated images produced using optimal transport to those generated by other standard interpolation methods. The novel optimal transport--based approach presented indeed gives smoother image interpolations, suggesting that viewing the embeddings as point clouds (rather than as matrices) better reflects and leverages the geometry of the embedding space.

Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusion

TL;DR

The paper reframes CLIP prompt embeddings as point clouds in Wasserstein space to study their geometry within Stable Diffusion. By exploiting permutation-invariance of cross-attention, embeddings are interpolated along optimal-transport geodesics, producing smoother intermediate images than standard linear or random couplings. The approach is validated on a large set of prompt pairs from Crisscrossed Captions, using LPIPS-based Perceptual Path Length to quantify path smoothness, with OT dominating CLIP and random methods, especially for more similar prompts. The work highlights a geometry-aware view of embedding spaces that improves image interpolation and suggests directions for richer metrics and broader applicability to newer diffusion models and multi-modal token spaces.

Abstract

It can be shown that Stable Diffusion has a permutation-invariance property with respect to the rows of Contrastive Language-Image Pretraining (CLIP) embedding matrices. This inspired the novel observation that these embeddings can naturally be interpreted as point clouds in a Wasserstein space rather than as matrices in a Euclidean space. This perspective opens up new possibilities for understanding the geometry of embedding space. For example, when interpolating between embeddings of two distinct prompts, we propose reframing the interpolation problem as an optimal transport problem. By solving this optimal transport problem, we compute a shortest path (or geodesic) between embeddings that captures a more natural and geometrically smooth transition through the embedding space. This results in smoother and more coherent intermediate (interpolated) images when rendered by the Stable Diffusion generative model. We conduct experiments to investigate this effect, comparing the quality of interpolated images produced using optimal transport to those generated by other standard interpolation methods. The novel optimal transport--based approach presented indeed gives smoother image interpolations, suggesting that viewing the embeddings as point clouds (rather than as matrices) better reflects and leverages the geometry of the embedding space.

Paper Structure

This paper contains 11 sections, 6 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries and theoretical motivation
  3. Diffusion models, stable diffusion, and CLIP
  4. Permutation invariance
  5. Optimal transport and Wasserstein space
  6. Geometry-informed Image Interpolation Approach
  7. Experimental Design and Results
  8. Experimental setup
  9. Dataset
  10. Results
  11. Conclusions and Future Directions

Figures (3)

  • Figure 1: Embedding interpolation workflow where SD indicates use of the Stable Diffusion model to produce the associated image.
  • Figure 2: Image interpolations for each method across four selected prompt pairs of varying similarity.
  • Figure 3: PPL (left) and coupling cost (right) by interpolation method and similarity group.