GPN: Generative Point-based NeRF

TL;DR

The paper tackles the problem of incomplete scenes captured by real-world scanners by introducing Generative Point-based NeRF (GPN), which jointly reconstructs and repairs point clouds through a hypernetwork-based VAE that outputs NeRF weights conditioned on colored point clouds. It presents two frameworks: Generation, which reconstructs a high-resolution complete cloud, and Completion, which repairs missing regions while maintaining multi-view consistency via per-scene fine-tuning. The approach leverages voxel-based sampling with NeRFAcc and TriVol-inspired feature extraction to render and optimize at high quality with modest memory, achieving competitive results on ShapeNet for reconstruction, repair, and completion. The work enables color-consistent, high-fidelity implicit representations from sparse inputs and supports flexible completion and interpolation across object parts, with practical implications for industrial 3D reconstruction and editing.

Abstract

Scanning real-life scenes with modern registration devices typically gives incomplete point cloud representations, primarily due to the limitations of partial scanning, 3D occlusions, and dynamic light conditions. Recent works on processing incomplete point clouds have always focused on point cloud completion. However, these approaches do not ensure consistency between the completed point cloud and the captured images regarding color and geometry. We propose using Generative Point-based NeRF (GPN) to reconstruct and repair a partial cloud by fully utilizing the scanning images and the corresponding reconstructed cloud. The repaired point cloud can achieve multi-view consistency with the captured images at high spatial resolution. For the finetunes of a single scene, we optimize the global latent condition by incorporating an Auto-Decoder architecture while retaining multi-view consistency. As a result, the generated point clouds are smooth, plausible, and geometrically consistent with the partial scanning images. Extensive experiments on ShapeNet demonstrate that our works achieve competitive performances to the other state-of-the-art point cloud-based neural scene rendering and editing performances.

Figures (11)

• Figure 1: Flow diagram of the Generation Framework.
• Figure 2: Flow diagram of the Completion Framework.
• Figure 3: Novel object generation from different parts of objects. This figure shows the reconstruction of point clouds stitched together from different parts. The first row represents the point cloud parts from one object, while the first column represents parts from another. The remaining portion of the figure demonstrates the stitched reconstructions performed by the "Completion Framework", which inherits geometrical properties from different parts of different objects.
• Figure 4: Interpolations between different object parts. To perform interpolation between different objects, we cut the partial cloud from one object as a constraint and then perform interpolation between two partial clouds from two different objects. We can observe that the surface points of the generated cloud perform continuous deformation between two partial clouds.
• Figure 5: Completion Process. We simulate an incomplete scanning process and observe that the completed cloud geometry becomes more consistent with the captured frames, even if it does not cover the entire object. The completed surface mesh also has a more distinct texture after fine-tuning compared to the reconstructed result before fine-tuning.