Geometry-Editable and Appearance-Preserving Object Compositon

TL;DR

The paper addresses the challenge of General Object Composition by enabling geometry-editable edits while preserving fine-grained appearance. It introduces DGAD, a diffusion-based framework with a geometry-editable encoder that leverages CLIP/DINO semantic embeddings to implicitly capture object geometry, and a dense, gated appearance-preserving decoder that retrieves and spatially aligns appearance features from a reference network with the edited geometry. The approach achieves state-of-the-art performance across editability, appearance fidelity, and semantic consistency on large-scale benchmarks, validated through quantitative metrics and qualitative studies, plus a user study. This work enables more reliable and flexible AR/VR content creation and interactive image editing by tightly coupling geometry reasoning with appearance preservation in generative compositing.

Abstract

General object composition (GOC) aims to seamlessly integrate a target object into a background scene with desired geometric properties, while simultaneously preserving its fine-grained appearance details. Recent approaches derive semantic embeddings and integrate them into advanced diffusion models to enable geometry-editable generation. However, these highly compact embeddings encode only high-level semantic cues and inevitably discard fine-grained appearance details. We introduce a Disentangled Geometry-editable and Appearance-preserving Diffusion (DGAD) model that first leverages semantic embeddings to implicitly capture the desired geometric transformations and then employs a cross-attention retrieval mechanism to align fine-grained appearance features with the geometry-edited representation, facilitating both precise geometry editing and faithful appearance preservation in object composition. Specifically, DGAD builds on CLIP/DINO-derived and reference networks to extract semantic embeddings and appearance-preserving representations, which are then seamlessly integrated into the encoding and decoding pipelines in a disentangled manner. We first integrate the semantic embeddings into pre-trained diffusion models that exhibit strong spatial reasoning capabilities to implicitly capture object geometry, thereby facilitating flexible object manipulation and ensuring effective editability. Then, we design a dense cross-attention mechanism that leverages the implicitly learned object geometry to retrieve and spatially align appearance features with their corresponding regions, ensuring faithful appearance consistency. Extensive experiments on public benchmarks demonstrate the effectiveness of the proposed DGAD framework.

Figures (7)

• Figure 1: (a) Leverages compact semantic embeddings to enable object editability but fails to preserve appearance details. (b) Utilizes appearance features to retain visual fidelity, yet lacks editing capability. Unlike both, Our method implicitly learns the geometry-editable representation and explicitly aligns fine-grained appearance features with the geometry-edited representation, facilitating both precise geometry editing and faithful appearance preservation.
• Figure 2: The training process of the proposed Disentangled Geometry-editable and Appearance-preserving Diffusion (DGAD) is as follows (the inference process is similar but involves iterative denoising): It first leverages semantic embeddings to implicitly capture the desired geometric transformations, and then employs a cross-attention retrieval mechanism to align fine-grained appearance features with the geometry-edited representation, facilitating both precise geometry editing and faithful appearance preservation in object composition.
• Figure 3: Qualitative comparison with recent advanced methods. The results show that the proposed method can edit objects with desired geometric properties to align with the background scene while preserving their appearance details. For more detailed and comprehensive visualizations, please refer to the appendix \ref{['visual']}.
• Figure 4: Left half: Visualization of the implicitly captured geometric properties of the object. Right half: Qualitative comparisons of "Ours w/o dense CA" and "Ours", Using dense cross-attention effectively retrieves accurate appearance features, thereby promoting appearance preservation.
• Figure 5: Qualitative comparison with recent advanced methods. The images shown are sampled from the test dataset.