Points2NeRF: Generating Neural Radiance Fields from 3D point cloud

TL;DR

Points2NeRF introduces a hypernetwork-based autoencoder that converts 3D colored point clouds into NeRF weights, enabling conditioned and generative Neural Radiance Field representations without relying solely on 2D images. The approach uses a PointNet-like encoder to produce a latent code, which a decoder then maps to NeRF weights $\Theta$ for a target network $F_{\Theta}:(x,y,z,\theta,\psi)\rightarrow (r,g,b,\sigma)$, enabling view synthesis and later mesh extraction via marching cubes. Experiments on ShapeNet show that Points2NeRF yields competitive PSNR and enables high-quality mesh reconstruction, while also serving as a useful pre-training initializer that significantly speeds up NeRF training. The work highlights practical impact for robotics and graphics by enabling robust NeRFs from sparse point clouds and offering a generative, conditioned 3D representation with potential for interpolation in object space.

Abstract

Contemporary registration devices for 3D visual information, such as LIDARs and various depth cameras, capture data as 3D point clouds. In turn, such clouds are challenging to be processed due to their size and complexity. Existing methods address this problem by fitting a mesh to the point cloud and rendering it instead. This approach, however, leads to the reduced fidelity of the resulting visualization and misses color information of the objects crucial in computer graphics applications. In this work, we propose to mitigate this challenge by representing 3D objects as Neural Radiance Fields (NeRFs). We leverage a hypernetwork paradigm and train the model to take a 3D point cloud with the associated color values and return a NeRF network's weights that reconstruct 3D objects from input 2D images. Our method provides efficient 3D object representation and offers several advantages over the existing approaches, including the ability to condition NeRFs and improved generalization beyond objects seen in training. The latter we also confirmed in the results of our empirical evaluation.

Figures (7)

• Figure 1: Conversely to other approaches, our Points2NeRF method takes a unstructured 3D point cloud with colors as an input instead of 2D images. A straightforward substitution of the input data in the existing architectures is not feasible due to the sparsity of point cloud representation ( left image), which yields artifacts in reconstruction (baseline KNN-NeRF result in the middle). By extending an auto-encoder architecture with the hypernetwork paradigm, our Points2NeRF architecture can build an internal representation of complete 3D objects, which enhances the rendering quality of NeRFs ( right image).
• Figure 2: Our Points2NeRF approach takes a 3D point cloud with the associated color values and returns the weights of a NeRF network that reconstructs 3D objects with high fidelity and coherent coloring.
• Figure 3: Reconstructions of objects obtained by Points2NeRF.
• Figure 4: From NeRF representation, we can extract 3D mesh with color. Mesh representation of objects produced by Points2NeRF.
• Figure 5: Object generated by Generative Points2NeRF.