CEI-3D: Collaborative Explicit-Implicit 3D Reconstruction for Realistic and Fine-Grained Object Editing

Abstract

Existing 3D editing methods often produce unrealistic and unrefined results due to the deeply integrated nature of their reconstruction networks. To address the challenge, this paper introduces CEI-3D, an editing-oriented reconstruction pipeline designed to facilitate realistic and fine-grained editing. Specifically, we propose a collaborative explicit-implicit reconstruction approach, which represents the target object using an implicit SDF network and a differentially sampled, locally controllable set of handler points. The implicit network provides a smooth and continuous geometry prior, while the explicit handler points offer localized control, enabling mutual guidance between the global 3D structure and user-specified local editing regions. To independently control each attribute of the handler points, we design a physical properties disentangling module to decouple the color of the handler points into separate physical properties. We also propose a dual-diffuse-albedo network in this module to process the edited and non-edited regions through separate branches, thereby preventing undesired interference from editing operations. Building on the reconstructed collaborative explicit-implicit representation with disentangled properties, we introduce a spatial-aware editing module that enables part-wise adjustment of relevant handler points. This module employs a cross-view propagation-based 3D segmentation strategy, which helps users to edit the specified physical attributes of a target part efficiently. Extensive experiments on both real and synthetic datasets demonstrate that our approach achieves more realistic and fine-grained editing results than the state-of-the-art (SOTA) methods while requiring less editing time. Our code is available on https://github.com/shiyue001/CEI-3D.

Figures (10)

• Figure 1: Comparison with popular methods. Our approach shows better editing results. It maintains the photo-realism and fidelity to the user scribble when synthesized from different viewpoints. Not only is our PSNR (dB) improvement encouraging, but the reduction in edit time is impressive. Concretely, editing times on this example of EditNeRF liu2021editing, NeuMesh yang2022neumesh, and Ours are 45s, 1h, and 14s.
• Figure 2: The main ideas and the capabilities comparison. (a) Unlike traditional explicit or implicit methods, our approach facilitates editing by introducing flexible handler points and disentangling physical properties. (b) In comparison, our approach satisfies properties that are desirable for editing.
• Figure 3: Pipeline of CEI-3D. Training Phase. Given images and camera parameters, we first train an implicit surface network using an SDF MLP, shown on the top left of the diagram. To allow for flexible user control, we construct a handler point set $\mathcal{H}$ by sampling points on the implicit surface in a differential manner. Then, the combination of implicit surface and explicit handler points constitutes the collaborative 3D representation. To achieve controllability at the attribute level, we disentangle physical properties by optimizing learnable material parameters, lighting parameters, and diffuse branch $F_{at}$ for non-edited regions in the DDA network. Editing Phase. Given the user scribble, we find points to be modified and update the edited branch $F_{ae}$ of the DDA network using the target attribute values of specified points. After network parameter optimization, the edited results of any view can be rendered.
• Figure 4: Appearance editing comparisons. Qualitative comparison of our method with EditNeRF and NeuMesh for pattern painting edits on synthetic and real datasets. From left to right, we show the source image, user scribble, and novel-view rendering of edited results for each method. It can be observed that our approach provides consistent pattern edits on all of the examples.
• Figure 5: Boxplot illustration of user study. Our method demonstrates superior performance (high means) and greater stability across test objects (narrow interquartile range) in both editing accuracy and realism.