Exploration and Improvement of Nerf-based 3D Scene Editing Techniques

TL;DR

The paper surveys NeRF-based 3D scene editing, highlighting the high computational cost as a bottleneck for intuitive editing. It reviews early editing methods that decouple geometry and appearance via latent codes and conditional fields (e.g., Edit-NeRF, object-NeRF, NeuMesh, ARF). It covers generalizable editing via GaN-based 3D-aware GANs, diffusion-based multimodal editing, and 4D generation by combining multiple models, demonstrating progress toward real-time, multi-view-consistent editing and animation. Light/shadow editing via inverse rendering and microfacet-based approaches is discussed to improve relighting and detailed rendering, with open questions about scaling to large scenes and achieving efficiency.

Abstract

NeRF's high-quality scene synthesis capability was quickly accepted by scholars in the years after it was proposed, and significant progress has been made in 3D scene representation and synthesis. However, the high computational cost limits intuitive and efficient editing of scenes, making NeRF's development in the scene editing field facing many challenges. This paper reviews the preliminary explorations of scholars on NeRF in the scene or object editing field in recent years, mainly changing the shape and texture of scenes or objects in new synthesized scenes; through the combination of residual models such as GaN and Transformer with NeRF, the generalization ability of NeRF scene editing has been further expanded, including realizing real-time new perspective editing feedback, multimodal editing of text synthesized 3D scenes, 4D synthesis performance, and in-depth exploration in light and shadow editing, initially achieving optimization of indirect touch editing and detail representation in complex scenes. Currently, most NeRF editing methods focus on the touch points and materials of indirect points, but when dealing with more complex or larger 3D scenes, it is difficult to balance accuracy, breadth, efficiency, and quality. Overcoming these challenges may become the direction of future NeRF 3D scene editing technology.