Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion

TL;DR

The paper tackles instruction-guided editing of 4D scenes, a problem plagued by temporal and cross-view inconsistency when using traditional 2D diffusion priors. It introduces Instruct 4D-to-4D, which treats a 4D scene as a collection of pseudo-3D views and decouples editing into temporal-consistent pseudo-view edits and pseudo-3D application via distillation from Instruct-Pix2Pix. Key contributions include an anchor-aware IP2P with batched processing, an optical-flow guided sliding window for long sequences, depth-based pseudo-view propagation, and an iterative NeRF-fitting pipeline that updates the edited dataset until convergence. The approach yields sharper, more detailed, and 4D-consistent edits in both monocular and multi-camera settings, significantly outperforming a na"ive IN2N-4D baseline and enabling practical 4D scene editing with improved efficiency.

Abstract

This paper proposes Instruct 4D-to-4D that achieves 4D awareness and spatial-temporal consistency for 2D diffusion models to generate high-quality instruction-guided dynamic scene editing results. Traditional applications of 2D diffusion models in dynamic scene editing often result in inconsistency, primarily due to their inherent frame-by-frame editing methodology. Addressing the complexities of extending instruction-guided editing to 4D, our key insight is to treat a 4D scene as a pseudo-3D scene, decoupled into two sub-problems: achieving temporal consistency in video editing and applying these edits to the pseudo-3D scene. Following this, we first enhance the Instruct-Pix2Pix (IP2P) model with an anchor-aware attention module for batch processing and consistent editing. Additionally, we integrate optical flow-guided appearance propagation in a sliding window fashion for more precise frame-to-frame editing and incorporate depth-based projection to manage the extensive data of pseudo-3D scenes, followed by iterative editing to achieve convergence. We extensively evaluate our approach in various scenes and editing instructions, and demonstrate that it achieves spatially and temporally consistent editing results, with significantly enhanced detail and sharpness over the prior art. Notably, Instruct 4D-to-4D is general and applicable to both monocular and challenging multi-camera scenes. Code and more results are available at immortalco.github.io/Instruct-4D-to-4D.

Figures (8)

• Figure 1: Our Instruct 4D-to-4D edits 4D scenes as pseudo-3D scenes with 2D diffusion, achieving much sharper results with detailed textures across a variety of editing tasks and scenes. Notably, Instruct 4D-to-4D generates realistic and 4D consistent editing results in both monocular scenes and challenging multi-camera indoor scenes. Please refer to the supplementary video for additional visualization.
• Figure 2: Our Instruct 4D-to-4D edits a 4D scene by regarding it as a pseudo-3D scene with multiple pseudo-views, and then editing these pseudo-views in an iterative key frame-based pipeline. (a) Our pipeline edits the 4D scene by iteratively generating a fully edited dataset used to fit 4D NeRF. In each iteration, we first (b) edit each key pseudo-view through optical flow propagation and IP2P inpainting and repainting, and then (c) edit other pseudo-views by aggregating propagated results from both previous frames through optical flow, and the key pseudo-views at current frame through depth-based warping.
• Figure 3: Generation results show that our augmented IP2P achieves consistency within a batch via our anchor-aware attention module, and achieves consistency between different batches via the same anchor shared across batches. The white bounding box shows the most noticeable part of inconsistency.
• Figure 4: Qualitative results on various scenes demonstrate that our Instruct 4D-to-4D generates high-qualify editing results in style transfer tasks on various scenes. The edited scenes are well-consistent with the instructed style, showing bright colors and natural textures.
• Figure 5: Qualitative results on mochi-high-five scene in DyCheck dataset show that our Instruct 4D-to-4D achieves high-quality editing results over various editing instructions in the monocular scene. Our Instruct 4D-to-4D can even achieve consistent editing with complicated textures, e.g., in the Tiger editing, while baseline IN2N-4D generates blurred results with lots of artifacts.