Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates

TL;DR

This work tackles the problem of editing 3D scenes represented by neural radiance fields (NeRFs) through language-driven manipulation, enabling insertion and removal of objects without requiring precise geometry priors. The authors propose a two-stage approach: (1) fine-tune a text-to-image diffusion model to generate object-background blended views, and (2) progressively update the background NeRF using a pose-conditioned dataset updates strategy that introduces the object from nearby views to farther views, ensuring view-consistent rendering. A key contribution is the pose-conditioned data scheduler, which stabilizes NeRF training and enables both insertion and removal by alternating between 2D image synthesis and 3D reconstruction. Experimental results show photorealistic, view-consistent edits and outperforming state-of-the-art NeRF editing baselines on both qualitative and quantitative metrics, highlighting the method’s potential for flexible, user-friendly 3D scene editing without explicit geometry priors.

Abstract

Neural radiance field is an emerging rendering method that generates high-quality multi-view consistent images from a neural scene representation and volume rendering. Although neural radiance field-based techniques are robust for scene reconstruction, their ability to add or remove objects remains limited. This paper proposes a new language-driven approach for object manipulation with neural radiance fields through dataset updates. Specifically, to insert a new foreground object represented by a set of multi-view images into a background radiance field, we use a text-to-image diffusion model to learn and generate combined images that fuse the object of interest into the given background across views. These combined images are then used for refining the background radiance field so that we can render view-consistent images containing both the object and the background. To ensure view consistency, we propose a dataset updates strategy that prioritizes radiance field training with camera views close to the already-trained views prior to propagating the training to remaining views. We show that under the same dataset updates strategy, we can easily adapt our method for object insertion using data from text-to-3D models as well as object removal. Experimental results show that our method generates photorealistic images of the edited scenes, and outperforms state-of-the-art methods in 3D reconstruction and neural radiance field blending.

Figures (5)

• Figure 1: Object insertion. We propose a language-driven method for view-consistent 3D object insertion into a background NeRF scene. Given an object defined by a set of multi-view images, our method generates plausible text-guided insertion results that require geometry manipulation of the original background NeRF.
• Figure 2: Overview of our pipeline. We customize and fine-tune a text-to-image diffusion model for view synthesis in an inpainting manner (left). We then apply the model to progressively fuse an object into background views to update a background NeRF (right). The process of view synthesis and NeRF updating is performed repeatedly. Views generated by the diffusion model are added to an on-going dataset to strengthen the NeRF. In return, the NeRF renders color hints for the diffusion model to create new views.
• Figure 3: Qualitative results of object insertion. Inputs include multi-view object/background images, a 3D bounding box where the object is inserted in, and a text prompt. Note that some baselines use parts of the inputs due to the nature of their techniques.
• Figure 4: Qualitative results of object removal where we show the importance of pseudo ground-truth background (BG) in generating view-consistent editing results.
• Figure 5: Ablation study results. Each row shows the results of a variant of our pipeline. The left and right columns include images of two different views of an edited scene. For each column, from left to right are the results of increasing training steps, where the most left image in each column is an early-stage result and the most right is the final output. The left column is a view near the starting view, which converges faster than the right column from a farther view (except for the variant in (c) where both views converge equally fast).