VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization

TL;DR

VQ-NeRF addresses the challenge of discretizing materials in neural reflectance fields for 3D scenes. It introduces a two-branch architecture where a continuous branch estimates BRDF factors and a discrete branch quantizes these factors via vector quantization to yield a material segmentation map, enabling straightforward editing. A dropout-based ranking strategy automatically selects the number of materials, and two-branch joint training suppresses noise while enhancing segmentation quality. Evaluations on CG and real-world data show improved reconstruction, material decomposition, relighting, and interactive material editing compared to prior methods, marking the first effective discrete material editing in 3D scenes.

Abstract

We propose VQ-NeRF, a two-branch neural network model that incorporates Vector Quantization (VQ) to decompose and edit reflectance fields in 3D scenes. Conventional neural reflectance fields use only continuous representations to model 3D scenes, despite the fact that objects are typically composed of discrete materials in reality. This lack of discretization can result in noisy material decomposition and complicated material editing. To address these limitations, our model consists of a continuous branch and a discrete branch. The continuous branch follows the conventional pipeline to predict decomposed materials, while the discrete branch uses the VQ mechanism to quantize continuous materials into individual ones. By discretizing the materials, our model can reduce noise in the decomposition process and generate a segmentation map of discrete materials. Specific materials can be easily selected for further editing by clicking on the corresponding area of the segmentation outcomes. Additionally, we propose a dropout-based VQ codeword ranking strategy to predict the number of materials in a scene, which reduces redundancy in the material segmentation process. To improve usability, we also develop an interactive interface to further assist material editing. We evaluate our model on both computer-generated and real-world scenes, demonstrating its superior performance. To the best of our knowledge, our model is the first to enable discrete material editing in 3D scenes.

Figures (17)

• Figure 1: Conventional neural reflectance decomposition methods (upper row) often predict noisy BRDF attributes for individual materials due to the absence of material discretization. This continuous representation also presents challenges for specific material editing. In contrast, our VQ-NeRF approach (lower row) incorporates the VQ mechanism to discretize reflectance decomposition, which suppresses prediction noise and facilitates material editing.
• Figure 2: We propose VQ-NeRF, which incorporates the VQ mechanism to discretize reflectance decomposition. This enables efficient and view-consistent material selection and editing.
• Figure 3: The pipeline of our VQ-NeRF, the outputs are marked by asterisks (*). We first take multi-view posed images as inputs and use a NeRF model (gray part) to reconstruct the scene geometry. Next, we apply a two-branch network for reflectance decomposition and material discretization. The continuous branch (green part) predicts the decomposed BRDF attributes, including diffuse, specular, and roughness, while the discrete branch (red part) uses the VQ mechanism to discretize reflectance factors. After optimization, a material segmentation map is generated, which enables us to easily select specific materials for editing.
• Figure 4: Our UI for interactive material selection and editing. The reconstructed image and segmentation map of an arbitrary view are presented in (a), (b) and (c). Users can click in (b) to specify the editing area, and assign the target material in (e). The lighting of the scene can also be adjusted in (f). After configuring all the settings, the user can start the model by clicking the 'Edit!' button. The edited results are shown in (d) for visualization.
• Figure 5: Reconstruction and reflectance decomposition results on the CG dataset provided by previous methods. Obviously, the reflectance factors predicted by our model exhibit the most proper color and the least noise.