Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LatentEditor: Text Driven Local Editing of 3D Scenes

Umar Khalid, Hasan Iqbal, Nazmul Karim, Jing Hua, Chen Chen

TL;DR

LatentEditor tackles the challenge of editing real-world NeRF scenes by enabling text-driven, localized modifications in a latent diffusion space. The method embeds scenes into a Stable Diffusion latent space, trains the NeRF directly in that space, and employs a delta module to produce 2D masks that constrain edits to targeted regions, guided by InstructPix2Pix. A refining adapter with self-attention maintains consistency between rendered latents and the original scene, while DDIM-based editing updates latent variables within the mask. Empirical results on LLFF, IN2N, NeRFStudio, and NeRF-Art show faster editing (5–8× fewer iterations) and improved CLIP-based fidelity and perceptual quality compared to IN2N and NeRF-Art, demonstrating effective, text-driven local 3D editing with preserved background and multi-attribute capabilities.

Abstract

While neural fields have made significant strides in view synthesis and scene reconstruction, editing them poses a formidable challenge due to their implicit encoding of geometry and texture information from multi-view inputs. In this paper, we introduce \textsc{LatentEditor}, an innovative framework designed to empower users with the ability to perform precise and locally controlled editing of neural fields using text prompts. Leveraging denoising diffusion models, we successfully embed real-world scenes into the latent space, resulting in a faster and more adaptable NeRF backbone for editing compared to traditional methods. To enhance editing precision, we introduce a delta score to calculate the 2D mask in the latent space that serves as a guide for local modifications while preserving irrelevant regions. Our novel pixel-level scoring approach harnesses the power of InstructPix2Pix (IP2P) to discern the disparity between IP2P conditional and unconditional noise predictions in the latent space. The edited latents conditioned on the 2D masks are then iteratively updated in the training set to achieve 3D local editing. Our approach achieves faster editing speeds and superior output quality compared to existing 3D editing models, bridging the gap between textual instructions and high-quality 3D scene editing in latent space. We show the superiority of our approach on four benchmark 3D datasets, LLFF, IN2N, NeRFStudio and NeRF-Art. Project Page: https://latenteditor.github.io/

LatentEditor: Text Driven Local Editing of 3D Scenes

TL;DR

LatentEditor tackles the challenge of editing real-world NeRF scenes by enabling text-driven, localized modifications in a latent diffusion space. The method embeds scenes into a Stable Diffusion latent space, trains the NeRF directly in that space, and employs a delta module to produce 2D masks that constrain edits to targeted regions, guided by InstructPix2Pix. A refining adapter with self-attention maintains consistency between rendered latents and the original scene, while DDIM-based editing updates latent variables within the mask. Empirical results on LLFF, IN2N, NeRFStudio, and NeRF-Art show faster editing (5–8× fewer iterations) and improved CLIP-based fidelity and perceptual quality compared to IN2N and NeRF-Art, demonstrating effective, text-driven local 3D editing with preserved background and multi-attribute capabilities.

Abstract

While neural fields have made significant strides in view synthesis and scene reconstruction, editing them poses a formidable challenge due to their implicit encoding of geometry and texture information from multi-view inputs. In this paper, we introduce \textsc{LatentEditor}, an innovative framework designed to empower users with the ability to perform precise and locally controlled editing of neural fields using text prompts. Leveraging denoising diffusion models, we successfully embed real-world scenes into the latent space, resulting in a faster and more adaptable NeRF backbone for editing compared to traditional methods. To enhance editing precision, we introduce a delta score to calculate the 2D mask in the latent space that serves as a guide for local modifications while preserving irrelevant regions. Our novel pixel-level scoring approach harnesses the power of InstructPix2Pix (IP2P) to discern the disparity between IP2P conditional and unconditional noise predictions in the latent space. The edited latents conditioned on the 2D masks are then iteratively updated in the training set to achieve 3D local editing. Our approach achieves faster editing speeds and superior output quality compared to existing 3D editing models, bridging the gap between textual instructions and high-quality 3D scene editing in latent space. We show the superiority of our approach on four benchmark 3D datasets, LLFF, IN2N, NeRFStudio and NeRF-Art. Project Page: https://latenteditor.github.io/
Paper Structure (13 sections, 17 equations, 10 figures, 2 tables)

This paper contains 13 sections, 17 equations, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Framework
  4. Background
  5. Method
  6. Latent Training.
  7. Latent Editing: An Application of Latent-NeRF
  8. Experimental Analysis
  9. Results
  10. Ablations
  11. Limitations
  12. Conclusion
  13. Acknowledgement

Figures (10)

  • Figure 1: Our proposed method LatentEditor enables text-based NeRF editing (e.g. color, attributes, style, etc.) by instilling both spatial and 3D awareness into image diffusion models. It can be observed in all the results that the background is intact with the color change or style transfer textual prompts.
  • Figure 2: Local Editing Challenge: Comparative analysis of local editing capabilities between our LatentEditor method and other text-driven 3D NeRF editing approaches, specifically IN2N haque2023instruct and NeRF-Art wang2023nerf. Our approach preserves the background seamlessly under both editing prompts.
  • Figure 3: Overall pipeline of LatentEditor. (a) We initialize the NeRF model within the latent domain, guided by the latent features of the original dataset. Our refinement adapter mitigates the misalignment in the latent space and encompasses a trainable adapter with residual and self-attention mechanisms. (b) Upon initialization, LatentEditor iteratively refines the model within the latent space for a predetermined number of iterations, while consistently updating the training set with the edited latents, $Z_e$. (c) The Delta Module is adept at interpreting prompts and produces the mask for targeted editing. An RGB image can be obtained by feeding the edited latent to the stable diffusion (SD) rombach2022high decoder $\mathcal{D}$ whereas $\mathcal{E}$ represents SD rombach2022high encoder.
  • Figure 4: Qualitative Results. The visual results of our approach, when contrasted with the baseline IN2N haque2023instruct across two distinct scenes, distinctly demonstrate that LatentEditor excels in accurately pinpointing the pertinent region, executing faithful text edits, and averting undesired alterations. These achievements prove challenging for the baseline method haque2023instruct to replicate effectively as IN2N haque2023instruct also changes the background objects' color to blue given that the editing prompt, "Make his hair red and give him blue jacket" only wants the jacket color to be changed.
  • Figure 5: Style Transfer Comparison. We present a visual representation for stylization editing, comparing our results with those obtained using NeRF-Art wang2023nerf and IN2N haque2023instruct. It can be observed that LatentEditor keeps the background intact while transferring the style of an object.
  • ...and 5 more figures