Edit3r: Instant 3D Scene Editing from Sparse Unposed Images

TL;DR

Edit3r tackles the challenge of fast, instruction-driven 3D scene editing from unposed, edited 2D images by delivering a single-pass, feed-forward pipeline that predicts 3D Gaussian splats aligned to a text prompt. It introduces a SAM2-based recoloring strategy for view-consistent supervision and an asymmetric input design to robustly fuse edited and unedited views, avoiding per-scene optimization. A dedicated DL3DV-Edit-Bench benchmark enables standardized evaluation of edit effectiveness and 3D consistency across diverse scenes and edit types. Empirical results show Edit3r outperforms optimization-based and prior feed-forward baselines in semantic alignment and cross-view realism while achieving real-time inference, signaling strong potential for interactive 3D editing workflows.

Abstract

We present Edit3r, a feed-forward framework that reconstructs and edits 3D scenes in a single pass from unposed, view-inconsistent, instruction-edited images. Unlike prior methods requiring per-scene optimization, Edit3r directly predicts instruction-aligned 3D edits, enabling fast and photorealistic rendering without optimization or pose estimation. A key challenge in training such a model lies in the absence of multi-view consistent edited images for supervision. We address this with (i) a SAM2-based recoloring strategy that generates reliable, cross-view-consistent supervision, and (ii) an asymmetric input strategy that pairs a recolored reference view with raw auxiliary views, encouraging the network to fuse and align disparate observations. At inference, our model effectively handles images edited by 2D methods such as InstructPix2Pix, despite not being exposed to such edits during training. For large-scale quantitative evaluation, we introduce DL3DV-Edit-Bench, a benchmark built on the DL3DV test split, featuring 20 diverse scenes, 4 edit types and 100 edits in total. Comprehensive quantitative and qualitative results show that Edit3r achieves superior semantic alignment and enhanced 3D consistency compared to recent baselines, while operating at significantly higher inference speed, making it promising for real-time 3D editing applications.

Figures (6)

• Figure 1: Overview.Edit3r takes view-inconsistent, 2D edited images as input and generates a view-consistent 3D scene for novel view synthesis in only 0.5 seconds. It supports diverse editing tasks, including insertion, removal, color changes, and weather changes, etc.
• Figure 2: Training pipeline of Edit3r. Green-framed images are input views and purple-framed images are supervision views (ground-truth and rendered results). We first apply SAM2-based recoloring to both inputs and supervision views, then feed an asymmetric pair (one recolored view and one original view) into Edit3r to predict a 3D Gaussian scene, which is rendered and supervised with reconstruction losses against the recolored supervision views (right). In parallel, a frozen LRM reconstructs the scene from the original inputs and provides 3D supervision via geometry losses that regularize Edit3r’s Gaussian predictions in 3D space.
• Figure 3: Example of SAM2-based recoloring process.
• Figure 4: Example of inference-time editing. (a) First input view, its edited result, and the corresponding single-view Gaussian rendering. (b) Second input view, its edited result, and the corresponding single-view Gaussian rendering. (c) Final rendering obtained by combining Gaussians from both views. Prompt: Add a cactus garden.
• Figure 5: Qualitative comparison among four methods. The first row shows the original scene and the editing prompt; the remaining four rows show the results, each paired with an additional view.