IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models

TL;DR

This work proposes a heuristic bottleneck constraint based on a novel relative perceptual path diversity score that automatically controls the bottleneck size and balances the diversity along the path with its directness and proposes a perceptually-uniform sampling technique that enables visually smooth changes between the interpolated images.

Abstract

We present a diffusion-based image morphing approach with perceptually-uniform sampling (IMPUS) that produces smooth, direct and realistic interpolations given an image pair. The embeddings of two images may lie on distinct conditioned distributions of a latent diffusion model, especially when they have significant semantic difference. To bridge this gap, we interpolate in the locally linear and continuous text embedding space and Gaussian latent space. We first optimize the endpoint text embeddings and then map the images to the latent space using a probability flow ODE. Unlike existing work that takes an indirect morphing path, we show that the model adaptation yields a direct path and suppresses ghosting artifacts in the interpolated images. To achieve this, we propose a heuristic bottleneck constraint based on a novel relative perceptual path diversity score that automatically controls the bottleneck size and balances the diversity along the path with its directness. We also propose a perceptually-uniform sampling technique that enables visually smooth changes between the interpolated images. Extensive experiments validate that our IMPUS can achieve smooth, direct, and realistic image morphing and is adaptable to several other generative tasks.

Figures (27)

• Figure 1: Image Morphing with IMPUS. IMPUS achieves smooth, real, and direct interpolation between two images with perceptually-uniform transition. More results in Sec. \ref{['sec:experiments']} and Appendix.
• Figure 2: Compared with interpolation_icmlw2023, our method achieves better sample consistency and quality. See Appendix for more trajectories to see comparison in smoothness.
• Figure 3: Benchmark examples.
• Figure 4: Ablation studies under LoRA rank, unconditional noise estimates, and sampling methods. LoRA rank: (a)-(f) intra-class on flowers with $\gamma=0.44$, $d_{\text{CLIP}}=0.15$, (g)-(i) inter-class on beetle and car with $\gamma=0.39$, $d_{\text{CLIP}}=0.37$ , (j)-(l) inter-class on lion and camel with $\gamma=0.49$, $d_{\text{CLIP}}=0.29$. Unconditional noise estimates: (m) only finetune on conditioning, (n) randomly discard conditioning during finetune, (o) separate LoRA for unconditional estimates. Sampling methods: (o)-(p) perceptually uniform sampling versus uniform space sampling.
• Figure 5: Model explainability with IMPUS (ResNet50).