Coupled Diffusion Sampling for Training-Free Multi-View Image Editing

TL;DR

The paper tackles multi-view consistency in diffusion-based edits without training new 3D models. It introduces coupled diffusion sampling, a simple inference-time mechanism that jointly guides a 2D editing model and a pre-trained multi-view diffusion model via a coupling energy $U(x, x') = -\frac{\lambda}{2}\|x - x'\|_2^2$ to align outputs while preserving each model's prior. The approach is validated on spatial editing, stylization, and relighting, showing improved view coherence, identity preservation, and competitive per-image metrics across diverse backbones. This coupling framework offers a practical, training-free route to high-quality, multi-view edits with potential extensions to video and other conditional diffusion tasks.

Abstract

We present an inference-time diffusion sampling method to perform multi-view consistent image editing using pre-trained 2D image editing models. These models can independently produce high-quality edits for each image in a set of multi-view images of a 3D scene or object, but they do not maintain consistency across views. Existing approaches typically address this by optimizing over explicit 3D representations, but they suffer from a lengthy optimization process and instability under sparse view settings. We propose an implicit 3D regularization approach by constraining the generated 2D image sequences to adhere to a pre-trained multi-view image distribution. This is achieved through coupled diffusion sampling, a simple diffusion sampling technique that concurrently samples two trajectories from both a multi-view image distribution and a 2D edited image distribution, using a coupling term to enforce the multi-view consistency among the generated images. We validate the effectiveness and generality of this framework on three distinct multi-view image editing tasks, demonstrating its applicability across various model architectures and highlighting its potential as a general solution for multi-view consistent editing.

Figures (16)

• Figure 1: Applications of coupled diffusion sampling. Our approach enables lifting off-the-shelf 2D editing models into multi-view by combining the sampling process of 2D diffusion models with multi-view diffusion models to produce view-consistent edits. Here we showcase example view-consistent results using a 2D spatial editing model, stylization, and text-based relighting.
• Figure 2: Limitations of baselines. Using a pre-trained image-to-multiview model conditioned on an edited image, can only be faithful to that single image but not the rest of the input views. On the other hand, editing each image individually with the 2D model produces highly inconsistent results. While prior work liu2022compositional proposes a method to compose diffusion models within the same domain, we find that their approach produces flickering results and cannot guarantee being faithful to the input views.
• Figure 3: Overview of the proposed coupled sampling method. Given two target statistical distributions modeled with diffusion models: (a) standard DDPM sampling generates two instances independently, using scores from each distribution, which leads to samples without spatial alignment; (b) in contrast, the proposed coupled DDPM sampling introduces coupling terms $\nabla U$ that pull the two sample paths together, producing spatially and semantically aligned outputs; and (c) as illustrated, samples from the standard DDPM sampling produce independent samples. In contrast, coupled sampling produces spatially aligned samples while each sample correctly remain within its distribution.
• Figure 4: Multi-view stylization. We show three examples of multi-view stylization of our method against the baselines. Prior work on combining diffusion models liu2022compositionalyilun2023reuse suffer from inconsistencies across frames. SDS based methods elad2023texture suffer from severe artifacts. Hunyuan 3D's results follow the prompt loosely when doing retexturing.
• Figure 5: Qualitative comparison on multi-view spatial editing. The baselines struggle in preserving the identity of the input, and produce flickering artifacts across edited frames, while our results achieve both editing targets and multi-view consistency.