DiffPop: Plausibility-Guided Object Placement Diffusion for Image Composition

TL;DR

DiffPop tackles plausible object placement for realistic image composition by learning a placement distribution over $\mathbf{x}=[s,V,h]$ with an unguided diffusion model and enforcing scene-level plausibility via a gradient-guided classifier. It introduces a two-stage training pipeline and a classifier-guided sampling procedure, using a structural plausibility classifier $C_s$ trained with human annotations to steer placements. Across Cityscapes-OP and OPA, DiffPop achieves higher plausibility and diversity than strong baselines and demonstrates usefulness for data augmentation and multi-object placement, aided by the Cityscapes-OP dataset. The work opens avenues for generalization to unseen categories and joint placement-harmonization with future extensions to more complex scene constraints.

Abstract

In this paper, we address the problem of plausible object placement for the challenging task of realistic image composition. We propose DiffPop, the first framework that utilizes plausibility-guided denoising diffusion probabilistic model to learn the scale and spatial relations among multiple objects and the corresponding scene image. First, we train an unguided diffusion model to directly learn the object placement parameters in a self-supervised manner. Then, we develop a human-in-the-loop pipeline which exploits human labeling on the diffusion-generated composite images to provide the weak supervision for training a structural plausibility classifier. The classifier is further used to guide the diffusion sampling process towards generating the plausible object placement. Experimental results verify the superiority of our method for producing plausible and diverse composite images on the new Cityscapes-OP dataset and the public OPA dataset, as well as demonstrate its potential in applications such as data augmentation and multi-object placement tasks. Our dataset and code will be released.

Figures (7)

• Figure 1: The training pipeline of our DiffPop, consisting of two stages. In Stage-1 (above), we train an unguided object placement denoising diffusion model on object scales and locations. In Stage-2 (below), we first utilize the generated scales and locations from above pre-trained diffusion model to form the corresponding transformation matrix, followed by applying object composition to obtain the composite images; then, we adopt the human-in-the-loop strategy and manually label composite images into positive and negative classes based on plausibility and realism of composite images. These labeled data are then used for training the structural plausibility classifier $\mathbf{C}_s$.
• Figure 2: The inference pipeline of our DiffPop. We sample a random placement $\mathbf{x}_T$ from the Gaussian noise at step $T$ and iteratively denoise it till step $0$ to obtain the desired placement $\mathbf{x}_0$. When sampling at step $t$, we take the gradient from the structural plausibility classifier $\mathbf{C}_s$ for guided generation towards scene-level structural coherence.
• Figure 3: The detailed illustration for biased sampling guided by the structural plausibility classifier $\mathbf{C}_s$. $\mathbf{C}_s$ takes the composite layouts as the input and outputs the probability of structural plausibility, which is further utilized in the guided sampling of object placement diffusion model at step $t-1$.
• Figure 4: Qualitative results of single object placement on Cityscapes-OP. Best viewed with zoom-in.
• Figure 5: Qualitative results of single object placement on OPA dataset. Best viewed with zoom-in.