3D-Fixup: Advancing Photo Editing with 3D Priors

TL;DR

3D-Fixup introduces a diffusion-based, feed-forward framework that enables 3D-aware image editing by learning from real-world video data and explicit 3D priors. A novel data generation pipeline constructs training triples using video frames, 3D reconstructions, and 3D transformations $\mathbf{T}$ to produce guidance $I_{guide}$ and masks $M_{guide}$ for training two networks, $f_{gen}$ and $f_{detail}$, with cross-attention to preserve identity. The method supports large out-of-plane rotations and translations, outperforms state-of-the-art baselines on LPIPS and FID, and runs efficiently at test-time, addressing prior limitations of optimization-based and 2D-only editing approaches. This work bridges 2D diffusion-based editing with 3D priors, enabling realistic, 3D-consistent edits in natural images and paving the way for scalable 3D-aware manipulation in practical applications.

Abstract

Despite significant advances in modeling image priors via diffusion models, 3D-aware image editing remains challenging, in part because the object is only specified via a single image. To tackle this challenge, we propose 3D-Fixup, a new framework for editing 2D images guided by learned 3D priors. The framework supports difficult editing situations such as object translation and 3D rotation. To achieve this, we leverage a training-based approach that harnesses the generative power of diffusion models. As video data naturally encodes real-world physical dynamics, we turn to video data for generating training data pairs, i.e., a source and a target frame. Rather than relying solely on a single trained model to infer transformations between source and target frames, we incorporate 3D guidance from an Image-to-3D model, which bridges this challenging task by explicitly projecting 2D information into 3D space. We design a data generation pipeline to ensure high-quality 3D guidance throughout training. Results show that by integrating these 3D priors, 3D-Fixup effectively supports complex, identity coherent 3D-aware edits, achieving high-quality results and advancing the application of diffusion models in realistic image manipulation. The code is provided at https://3dfixup.github.io/

Figures (8)

• Figure 1: Inference pipeline. We assume editing instructions (possibly converted from text prompts) are in the form of 3D operations like rotation and translation. Given a mask indicating the object to be edited, we first perform image-to-3D xu2024instantmesh to reconstruct the mesh. We then apply the user's desired 3D edit to obtain the 3D guidance. Here the 3D edit is visualized as the transformation between original mesh (pink wire-frame) and the edited mesh (cyan wire-frame). Finally, the model outputs the 3D aware editing result.
• Figure 2: Data pipeline: Overview. Given a video, we sample two frames, the source frame $I_{\mathrm{src}}$ and the target frame $I_{\mathrm{tgt}}$, using optical flow as a cue: we discard videos where the flow indicates little motion through the entire clip. Using Image-to-3D methods, we reconstruct a mesh for the desired object for both frames. We then estimate the 3D transformation $\mathbf{T}$ (see Figure \ref{['fig:2.overview.dp2']}) between the source frame mesh and the target frame mesh. Availability of the transformation $\mathbf{T}$ enables two ways to create the training data: (1) in "Transform Source", we paste the rendering of the transformed source mesh onto the target frame; (2) in "Transform Target", we paste the rendering of the target mesh onto the source frame. Data examples are shown in Figure \ref{['fig:2.dataexp']}.
• Figure 3: Data pipeline: Estimation of the 3D transformation $\mathbf{T}$. We estimate the 3D transformation $\mathbf{T}$ by leveraging correspondences between the source frame and the target frame. Given frames between the source frame and the target frame, we first perform tracking to obtain corresponding points. We then initialize the parameters for the 3D transformation $\mathbf{T}$ and use an optimization to improve $\mathbf{T}$: (1) We unproject the 2D correspondences on the source frame to 3D pointclouds and apply the current $\mathbf{T}$ to transform points to the target image; (2) we project points back to 2D and compare via an L2 loss with the 2D correspondences of the target frame.
• Figure 4: Examples of the training data. Given a video, we use the steps described in Figure \ref{['fig:2.overview.dp1']} to obtain the training data, i.e., the source image $I_{\mathrm{src}}$ and the target image $I_{\mathrm{tgt}}$. The guidance image is obtained via the developed data pipeline. The mask has three values: 0 indicates the hole created by the coarse edit and the model needs to inpaint by looking at the details of the reference; 0.5 refers to the rendering of the object; and 1.0 denotes the original background.
• Figure 5: Overview of the training pipeline. We develop a conditional diffusion model for 3D-aware image editing. It consists of two networks: $f_{\mathrm{gen}}$ and $f_{\mathrm{detail}}$. During training, given the inputs—target frame $I_{\mathrm{tgt}}$, 3D guidance $I_{\mathrm{guide}}$, mask $M_{\mathrm{guide}}$, and detail feature $F_t$—$f_{\mathrm{gen}}$ learns the reverse diffusion process to predict the noise $\epsilon$ and reconstruct $I_{\mathrm{tgt}}$. To better preserve identity and fine-grained details from the source image $I_{\mathrm{src}}$, $f_{\mathrm{detail}}$ takes as input the source image $I_{\mathrm{src}}$, its noisy counterpart $I_t$, and the mask $M_{\mathrm{guide}}$, and extracts detail features $F_t$. We apply cross-attention between $F_t$ and the intermediate features of $f_{\mathrm{gen}}$ to incorporate content and details from $I_{\mathrm{src}}$ during the reverse diffusion process.