Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models

TL;DR

Add-it is introduced, a training-free approach that extends diffusion models' attention mechanisms to incorporate information from three key sources: the scene image, the text prompt, and the generated image itself, maintaining structural consistency and fine details while ensuring natural object placement.

Abstract

Adding Object into images based on text instructions is a challenging task in semantic image editing, requiring a balance between preserving the original scene and seamlessly integrating the new object in a fitting location. Despite extensive efforts, existing models often struggle with this balance, particularly with finding a natural location for adding an object in complex scenes. We introduce Add-it, a training-free approach that extends diffusion models' attention mechanisms to incorporate information from three key sources: the scene image, the text prompt, and the generated image itself. Our weighted extended-attention mechanism maintains structural consistency and fine details while ensuring natural object placement. Without task-specific fine-tuning, Add-it achieves state-of-the-art results on both real and generated image insertion benchmarks, including our newly constructed "Additing Affordance Benchmark" for evaluating object placement plausibility, outperforming supervised methods. Human evaluations show that Add-it is preferred in over 80% of cases, and it also demonstrates improvements in various automated metrics.

Figures (19)

• Figure 1: Given an input image (left in each pair), either real (top row) or generated (mid row), along with a simple textual prompt describing an object to be added Add-it seamlessly adds the object to the image in a natural way. Add-it allows the step-by-step creation of complex scenes without the need for optimization or pre-training.
• Figure 2: Architecture outline: Given a tuple of source noise $X_{source}^T$, target noise $X_{target}^T$, and a text prompt $P_{target}$, we first apply Structure Transfer to inject the source image's structure into the target image. We then extend the self-attention blocks so that $X_{target}^T$ pulls keys and values from both $P_{target}$ and $X_{source}^T$, with each source weighted separately. Finally, we use Subject Guided Latent Blending to retain fine details from the source image.
• Figure 3: User Study results evaluated on the real images from the Emu Edit Benchmark.
• Figure 4: User Study results evaluated on the generated images from the Image Additing Benchmark.
• Figure 5: Qualitative Results from the Emu-Edit Benchmark. Unlike other methods, which fail to place the object in a plausible location, our method successfully achieves realistic object insertion.