GeoDiffuser: Geometry-Based Image Editing with Diffusion Models

TL;DR

GeoDiffuser presents a zero-shot method that unifies 2D and 3D image editing by embedding geometric transformations directly into the shared attention of diffusion models. It uses object segmentation and optional depth maps, inverts the input image, and optimizes latents and null-text embeddings under losses that preserve background and object identity while inpainting disoccluded regions. The approach, which does not require training and works with any attention-enabled diffusion model, demonstrates strong quantitative and perceptual performance against baselines for translation, rotation, scaling, and removal. This technique advances practical image editing by enabling precise geometric edits in natural scenes with realistic lighting, shadows, and inpainted backgrounds. The results indicate significant potential for robust, geometry-driven editing in real-world workflows while highlighting areas for improvement in foreground disocclusion handling for large 3D rotations.

Abstract

The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However, these methods are bespoke, imprecise, require additional information, or are limited to only 2D image edits. We present GeoDiffuser, a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images, for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input, we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation, 3D rotation, and removal. We present quantitative results, including a perceptual study, that shows how our approach is better than existing methods. Visit https://ivl.cs.brown.edu/research/geodiffuser.html for more information.

Figures (20)

• Figure 1: Our method adheres to the desired edit having the least Mean Distance and Warp Error compared to Dragon Diffusion, FreeDrag, Diffusion Self Guidance, and Diffusion Handles.
• Figure 2: General image editing framework using diffusion models. (a) DDIM Inversion: The process of obtaining noise trajectory $\{{}^{r}z_{0}, {}^{r}z_{1}, ......, {}^{r}z_{t}\}$ for the reference image song2021denoising. (b) General Editing Framework: The Reference Diffusion Process guides the Edit Diffusion Process to achieve the desired edit. In GeoDiffuser, we perform geometric 2D and 3D edits by transforming the shared attention layers leading to plausible edits that preserves object style, inpainting disoccluded background, and adding details (e.g., the car's shadow).
• Figure 2: Metric Ablations: Increasing the number of time steps for geometric attention sharing and adaptive optimization both improve the Mean Distance, Warp Error, and Clip Similarity score. Removing removal loss introduces duplication of objects and removing background preservation changes the scene background.
• Figure 3: (a) GeoDiffuser attention sharing mechanism that leverages the geometric transformation $\mathcal{F}(\cdot)$ transform the reference attention ${{}^{\mathcal{G}}Y_{ref}}$ to guide the edit attention layer. (b) Optimization Loss Functions that penalize the latents and text-embeddings to perform the desired geometric edit. The orange mask highlights the region to be inpainted in the optimization.
• Figure 4: We perform the same edit using prior works and compare with out work. We show the intended 3D edit in column 2 where we highlight the region to be inpainted with orange and the region foreground inpainting region with green. Our work GeoDiffuser best adheres to the intended edit and ensures preservation of the scene without requiring prompts. Diffusion Handles requires an inpainting model and a depth trained diffusion model to perform the same edit with prompts but still fails to preserve the appearance of the scene. FreeDrag is slow and does not adhere well to the edit. Dragon Diffusion and Diffusion Self Guidance do not preserve the appearance of the object and do not rotate objects accurately. Please see supplement for a detailed analysis of all prior works.