Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion

TL;DR

The paper tackles the challenge of inserting arbitrary foreground objects into diverse scenes in a way that respects scene affordances. It introduces SAM-FB, a large-scale dataset derived from SA-1B, and proposes Mask-Aware Dual Diffusion (MADD), a dual-stream diffusion model that jointly denoises RGB appearance and insertion masks conditioned on foreground, background, and position prompts. Key contributions include the dataset construction pipeline with automated quality control, a dual-branch UNet architecture, and an effective encoding of sparse and dense position prompts, achieving state-of-the-art quantitative results and strong generalization to in-the-wild images. The work enables realistic, affordance-aware image composition and supports automated dataset synthesis for broader object-scene insertion tasks.

Abstract

As a common image editing operation, image composition involves integrating foreground objects into background scenes. In this paper, we expand the application of the concept of Affordance from human-centered image composition tasks to a more general object-scene composition framework, addressing the complex interplay between foreground objects and background scenes. Following the principle of Affordance, we define the affordance-aware object insertion task, which aims to seamlessly insert any object into any scene with various position prompts. To address the limited data issue and incorporate this task, we constructed the SAM-FB dataset, which contains over 3 million examples across more than 3,000 object categories. Furthermore, we propose the Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream architecture to simultaneously denoise the RGB image and the insertion mask. By explicitly modeling the insertion mask in the diffusion process, MADD effectively facilitates the notion of affordance. Extensive experimental results show that our method outperforms the state-of-the-art methods and exhibits strong generalization performance on in-the-wild images. Please refer to our code on https://github.com/KaKituken/affordance-aware-any.

Figures (17)

• Figure 1: Given a foreground-background object-scene pair, our model can perform affordance-aware object insertion conditioning on different position prompts, including points, bounding boxes, masks, and even null prompts.
• Figure 2: Dataset construction pipeline for SAM-FB. The pipeline automatically converts any input image into a tetrad output through four stages, ensuring high-quality foreground and background retention via a rigorous data quality control process.
• Figure 3: Mask-aware Dual Diffusion Model (MADD). The RGB image feature $\mathbf{z}$ and object mask $\mathbf{m}$ are jointly denoised, conditioning on the embeddings of the foreground object $\mathbf{f}$, background object $\mathbf{b}$, and the prompt $\mathbf{p}$. (green: reverse process $t\rightarrow t\mathrm{-}1$)
• Figure 4: Qualitative results of MADD on the SAM-FB test set. Each row corresponds to one type of prompt, i.e., point, bounding box, mask, and null, respectively. Our MADD simultaneously predicts the RGB image and the object mask.
• Figure 5: We test ambiguous prompts (points and blank) on in-the-wild images. With point prompts, our model adjusts foreground properties for affordance-aware insertion, while it autonomously finds suitable positions when no prompt is given.