Compositional Image Decomposition with Diffusion Models

TL;DR

The approach, Decomp Diffusion, is an unsupervised method which, when given a single image, infers a set of different components in the image, each represented by a diffusion model, and demonstrates how components can capture different factors of the scene.

Abstract

Given an image of a natural scene, we are able to quickly decompose it into a set of components such as objects, lighting, shadows, and foreground. We can then envision a scene where we combine certain components with those from other images, for instance a set of objects from our bedroom and animals from a zoo under the lighting conditions of a forest, even if we have never encountered such a scene before. In this paper, we present a method to decompose an image into such compositional components. Our approach, Decomp Diffusion, is an unsupervised method which, when given a single image, infers a set of different components in the image, each represented by a diffusion model. We demonstrate how components can capture different factors of the scene, ranging from global scene descriptors like shadows or facial expression to local scene descriptors like constituent objects. We further illustrate how inferred factors can be flexibly composed, even with factors inferred from other models, to generate a variety of scenes sharply different than those seen in training time. Website and code at https://energy-based-model.github.io/decomp-diffusion.

Figures (32)

• Figure 1: Image Decomposition with Decomp Diffusion. Our unsupervised method can decompose an input image into both local factors, such as objects (Left), and global factors (Right), such as facial features. Additionally, our approach can combine the deduced factors for image reconstruction.
• Figure 2: Compositional Image Decomposition. We learn to decompose each input image into a set of denoising functions $\{\epsilon_\theta({\bm{x}}_i^t, t, | {\bm{z}}_k)\}$ representing $K$ factors, which can be composed to reconstruct the input.
• Figure 3: Global Factor Decomposition. Our method can enable global factor decomposition and reconstruction on CelebA-HQ (Left) and Virtual KITTI $2$ (Right). Note that discovered factors are labeled with posited factors.
• Figure 4: Reconstruction comparison. Our method can reconstruct input images with a high fidelity on CelebA-HQ.
• Figure 5: Global Factor Recombination. Recombination of inferred factors on Falcor3D and CelebA-HQ datasets. In Falcor3D (Left), we show image variations by varying inferred factors such as lighting intensity. In CelebA-HQ (Right), we recombine factors from two different inputs to generate novel face combinations.