ObjectAdd: Adding Objects into Image via a Training-Free Diffusion Modification Fashion

TL;DR

ObjectAdd addresses the challenge of inserting a user-specified object into a diffusion-generated image without retraining. It introduces three innovations: text embedding coalescence to avoid prompt interference, object-driven layout control via latent and attention injection, and prompted image inpainting with attention refocusing and object expansion to preserve the rest of the image. Quantitative and qualitative results against strong baselines show improved non-edited area consistency and high prompt-object alignment, though broader evaluation and newer diffusion backbones remain future work. The approach extends to real images through segmentation-based preprocessing and inversion, highlighting practical potential for interactive image editing with diffusion models.

Abstract

We introduce ObjectAdd, a training-free diffusion modification method to add user-expected objects into user-specified area. The motive of ObjectAdd stems from: first, describing everything in one prompt can be difficult, and second, users often need to add objects into the generated image. To accommodate with real world, our ObjectAdd maintains accurate image consistency after adding objects with technical innovations in: (1) embedding-level concatenation to ensure correct text embedding coalesce; (2) object-driven layout control with latent and attention injection to ensure objects accessing user-specified area; (3) prompted image inpainting in an attention refocusing & object expansion fashion to ensure rest of the image stays the same. With a text-prompted image, our ObjectAdd allows users to specify a box and an object, and achieves: (1) adding object inside the box area; (2) exact content outside the box area; (3) flawless fusion between the two areas

Figures (11)

• Figure 1: Image generation of ChatGPT-4. Image (a) is prompted by "A woman wearing glasses." while image (b) generates from subsequent "Now let her wear a hat." Despite the high-qualified image (a), adding a hat causes visual inconsistency in image (b).
• Figure 2: Generation comparison under settings of $\mathcal{P}$ = "A woman wearing glasses" and $\mathcal{W}$ = "A hat": (a) $\mathcal{P}$ prompted image $\mathcal{I}^0$; (b) the coalesce $\mathcal{E}_{\{\mathcal{P}, \mathcal{W}\}}$ prompted image $\bar{\mathcal{I}}^0$ after object-driven layout control; (c) $\bar{\mathcal{I}}^0$ with prompted image inpainting.
• Figure 3: Visualization under settings of $\mathcal{P}$ = "A woman wearing glasses" and $\mathcal{W}$ = "A hat". The user-drawn area is marked by red box: (a) coalesce $\mathcal{E}_{\{\mathcal{P}, \mathcal{W}\}}$ prompted cross-attention map for object "hat"; (b) clustering of the cross-attention map; (c) object-centralized area; (d) attention refocusing mask; (e) attention refocusing mask with mathematical morphology; (f) object expansion mask.
• Figure 4: Pre-processing of input real image.
• Figure 5: A visual comparison of different methods w.r.t. prompt $\mathcal{P}$ generated image and corresponding version of adding word $\mathcal{W}$ described object. Best view with zooming in.