PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor

TL;DR

PAIR Diffusion proposes an object-centric diffusion framework that treats images as collections of objects, each with a structure and appearance, enabling fine-grained edits without inversion. By extracting per-object panoptic structure and multi-layer appearance features (VGG and DINOv2) and conditioning diffusion models on these representations, the method supports appearance edits, shape edits, object addition, and object variation, applicable to both unconditional and foundational diffusion models. A multimodal classifier-free guidance mechanism jointly leverages structure, appearance, and text (or reference images) to achieve precise, controllable edits. Extensive qualitative and quantitative evaluations on LSUN, CelebA-HQ, and COCO demonstrate robust, localized edits with ablations showing the benefits of combined appearance features and CFG parameters. The work offers a versatile, practical object-level image editor that operates without inversion and can adapt to real-world editing scenarios.

Abstract

Generative image editing has recently witnessed extremely fast-paced growth. Some works use high-level conditioning such as text, while others use low-level conditioning. Nevertheless, most of them lack fine-grained control over the properties of the different objects present in the image, i.e. object-level image editing. In this work, we tackle the task by perceiving the images as an amalgamation of various objects and aim to control the properties of each object in a fine-grained manner. Out of these properties, we identify structure and appearance as the most intuitive to understand and useful for editing purposes. We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image. We show that having control over the properties of each object in an image leads to comprehensive editing capabilities. Our framework allows for various object-level editing operations on real images such as reference image-based appearance editing, free-form shape editing, adding objects, and variations. Thanks to our design, we do not require any inversion step. Additionally, we propose multimodal classifier-free guidance which enables editing images using both reference images and text when using our approach with foundational diffusion models. We validate the above claims by extensively evaluating our framework on both unconditional and foundational diffusion models. Please refer to https://vidit98.github.io/publication/conference-paper/pair_diff.html for code and model release.

Figures (17)

• Figure 1: PAIR Diffusion framework allows the appearance and structure editing of an image at the object level. Our framework is general and can enable object-level editing capabilities in both (a) unconditional diffusion models and (b) foundational diffusion models. Using our framework with a foundational diffusion model allows for comprehensive in-the-wild object-level editing capabilities.
• Figure 2: Overview of PAIR Diffusion. An image is seen as a composition of objects each defined by different properties like structure (shape and category), appearance, depth, etc. We focus on controlling structure and appearance. (a) During training, we extract structure and appearance information and train a diffusion model in a conditional manner. (b) At inference, the framework supports multiple editing operations by independently controlling the structure and appearance of any real image at the object level.
• Figure 3: Qualitative results for appearance editing. We can drive the edit with reference images as well as with text prompts.
• Figure 4: Qualitative results for adding objects and shape editing.
• Figure 5: Visual results for appearance control on LSUN bedroom. We show the results obtained with relevant baselines for editing the red area in the input image using the reference as a driver.