DreamCom: Finetuning Text-guided Inpainting Model for Image Composition

TL;DR

DreamCom tackles the problem of realism and fidelity in image composition by finetuning a text-guided image inpainting model on a few reference images for a given object, binding the object to a special token $[V]$. It introduces masked cross-attention and masked self-attention to shield the foreground generation from background interference while preserving foreground colors, enabling seamless insertion into new backgrounds without iterative segmentation. The method is evaluated on DreamEditBench and the newly constructed MureCom dataset, showing superior foreground fidelity (measured by DINO and CLIP-I) and competitive background preservation, with user studies favoring DreamCom. The work also contributes the MureCom dataset, facilitating multi-reference, foreground-aware image composition research. Overall, DreamCom offers a practical, annotation-light path to high-quality foreground incorporation in diffusion-based image editing.

Abstract

The goal of image composition is merging a foreground object into a background image to obtain a realistic composite image. Recently, generative composition methods are built on large pretrained diffusion models, due to their unprecedented image generation ability. However, they are weak in preserving the foreground object details. Inspired by recent text-to-image generation customized for certain object, we propose DreamCom by treating image composition as text-guided image inpainting customized for certain object. Specifically , we finetune pretrained text-guided image inpainting model based on a few reference images containing the same object, during which the text prompt contains a special token associated with this object. Then, given a new background, we can insert this object into the background with the text prompt containing the special token. In practice, the inserted object may be adversely affected by the background, so we propose masked attention mechanisms to avoid negative background interference. Experimental results on DreamEditBench and our contributed MureCom dataset show the outstanding performance of our DreamCom.

Figures (7)

• Figure 1: (a) Given a background image with a bounding box and multiple reference images of the foreground object, we aim to produce a composite image, in which the foreground object is seamlessly and reasonably placed in the bounding box. (b) We finetune a text-guided inpainting model using multiple reference images of the same object, in which the text prompt contains a special token $[V]$ associated with this object. All the cross-attention layers and some selected self-attention layers are masked.
• Figure 2: An example of spurious correspondence between background and text prompt. From left to right, we present the masked background image, reference images, and the generated image with the text prompt "A $[V]$ horse".
• Figure 3: An example of foreground color disrupted by the background image. From left to right, we present the masked background image, reference images, and the generated image with the text prompt "A $[V]$ guitar".
• Figure 4: Several examples from our MureCom dataset. In each example, we show a background image with a bounding box, and five reference images of one foreground object.
• Figure 5: The subfigure (a) demonstrates the effectiveness of masked self-attention (MCAttn). The subfigure (b) demonstrates the effectiveness of masked self-attention (MSAttn) in maintaining original color while preserving overall harmony.