EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selection

TL;DR

EditCast3D tackles the costly and inconsistent nature of applying 2D foundation-model edits to 3D scenes by proposing a non-iterative pipeline that edits a single first frame and propagates the change to the entire dataset via a first-frame guided video generation model. A subsequent view-selection stage based on pose-free 3D Gaussian Splatting filters for reconstruction-friendly views, which are then fed into a fast feedforward reconstruction (InstantSplat) to produce the final 3D asset. Key components include LoRA-based fine-tuning of the video model, mask handling for added objects, and a robust per-view quality score to ensure geometric fidelity. The results on standard 3D editing benchmarks show higher editing fidelity, better instruction adherence, and substantial efficiency gains, suggesting a scalable path for integrating foundation models into practical 3D editing pipelines.

Abstract

Recent advances in foundation models have driven remarkable progress in image editing, yet their extension to 3D editing remains underexplored. A natural approach is to replace the image editing modules in existing workflows with foundation models. However, their heavy computational demands and the restrictions and costs of closed-source APIs make plugging these models into existing iterative editing strategies impractical. To address this limitation, we propose EditCast3D, a pipeline that employs video generation foundation models to propagate edits from a single first frame across the entire dataset prior to reconstruction. While editing propagation enables dataset-level editing via video models, its consistency remains suboptimal for 3D reconstruction, where multi-view alignment is essential. To overcome this, EditCast3D introduces a view selection strategy that explicitly identifies consistent and reconstruction-friendly views and adopts feedforward reconstruction without requiring costly refinement. In combination, the pipeline both minimizes reliance on expensive image editing and mitigates prompt ambiguities that arise when applying foundation models independently across images. We evaluate EditCast3D on commonly used 3D editing datasets and compare it against state-of-the-art 3D editing baselines, demonstrating superior editing quality and high efficiency. These results establish EditCast3D as a scalable and general paradigm for integrating foundation models into 3D editing pipelines. The code is available at https://github.com/UNITES-Lab/EditCast3D

Figures (11)

• Figure 1: Results of EditCast3D. EditCast3D enables efficient, detailed, and precise editing of 3D scenes. It requires only a single edited image from the dataset produced by any image editing foundation model as the input. The edit is then propagated across the entire 3D scene. With the addition of the view selection mechanism, EditCast3D achieves high-quality reconstructions while demonstrating strong instruction-following and editing capability.
• Figure 2: Traditional image editing models like InstructPix2Pix brooks2023instructpix2pix provide relatively consistent edits across views, but their editing capabilities are limited. In contrast, recent foundation models enable more powerful edits, yet often introduce inconsistencies across views.
• Figure 3: Starting from a collection of input views, only the first frame is edited using an image editing foundation model. The edit is then propagated to the entire dataset via a video generation foundation model, enabling dataset-level editing with minimal computational cost. Since editing propagation alone may produce inconsistent views, EditCast3D further incorporates a view selection strategy that filters for consistent and reconstruction-friendly views. The selected views are finally fed into a feedforward 3D reconstruction module, producing edited 3D assets with high quality and consistency.
• Figure 4: Default image-to-video generation. Given a masked video sequence and masks, the model fills in the missing regions to synthesize complete frames.
• Figure 5: During training, a mask is applied to the regions requiring editing, and LoRAs are trained by reconstructing these masked regions given the first frame as reference. At inference time, the same mask is provided along with the edited first frame. The model then uses the first frame to guide the filling of the masked regions across subsequent frames, thereby achieving consistent video editing.