PointNeRF++: A multi-scale, point-based Neural Radiance Field

TL;DR

A simple representation that aggregates point clouds at multiple scale levels with sparse voxel grids at different resolutions, and adds a global voxel at the coarsest scale to help model areas without points, thus unifying ``classical'' and point-based NeRF formulations.

Abstract

Point clouds offer an attractive source of information to complement images in neural scene representations, especially when few images are available. Neural rendering methods based on point clouds do exist, but they do not perform well when the point cloud quality is low -- e.g., sparse or incomplete, which is often the case with real-world data. We overcome these problems with a simple representation that aggregates point clouds at multiple scale levels with sparse voxel grids at different resolutions. To deal with point cloud sparsity, we average across multiple scale levels -- but only among those that are valid, i.e., that have enough neighboring points in proximity to the ray of a pixel. To help model areas without points, we add a global voxel at the coarsest scale, thus unifying ``classical'' and point-based NeRF formulations. We validate our method on the NeRF Synthetic, ScanNet, and KITTI-360 datasets, outperforming the state of the art, with a significant gap compared to other NeRF-based methods, especially on more challenging scenes.

Figures (7)

• Figure 1: Teaser -- We introduce a novel volume-rendering framework to effectively leverage point clouds for Neural Radiance Fields. Our formulation aggregates points over multiple scales---including a global scale governing the entire scene, equivalent to the standard, point-agnostic NeRF. Our solution leads to much better novel-view synthesis in challenging real-world situations with sparse or incomplete point clouds. Here, we show example renderings from the KITTI-360 test set.
• Figure 2: Overview -- Given an input point cloud, we aggregate it over multi-scale voxel grids (\ref{['sec:multiscale']}). For clarity, we draw the voxel grids in 2D. We then perform volume rendering based on points, relying on feature vectors stored thereon, which we aggregate across multiple scales (\ref{['sec:volumerender']}). Importantly, when aggregating across scales, we only take into account 'valid' scales, i.e., those with nearby points---indicated with solid blue lines and illustrated as the two overlaid scales in the middle---naturally dealing with incomplete/sparse point clouds. The coarsest scale (the top row in the figure) is a single, global voxel, equivalent to standard NeRF---i.e., it is not point-based.
• Figure 3: Increasing coverage with multiple scales -- We illustrate our sparse, hierarchical representation at three granularity levels, including a single, global voxel (left). We also show three query points, with their respective neighborhoods (dotted circles) at each scale level---color-coded in blue if they have neighbouring features, and in orange if they do not. Our multi-scale approach naturally fills in empty regions, removing the need for failure-prone region-growing heuristics xu2022point. Drawn in 2D, for clarity.
• Figure 4: Examples on KITTI-360 -- We show novel-view renderings obtained with our method, 3D Gaussian splatting kerbl3Dgaussians (pink colored, 1-4 rows ) and PointNeRF xu2022point (green colored, 4-8 row) on a challenging outdoors dataset, using the same point clouds as input. Our approach provides significantly sharper renderings with more details, and better coverage in areas without points, where Gaussian Splatting and PointNeRF produce highly salient artifacts highlighted with red boxes.
• Figure 5: Examples on ScanNet -- PointNeRF fails to reconstruct the scene on regions where the point cloud is empty. Both our method and Gaussian Splatting are able to fill them in, but our approach produces cleaner results, with fewer artifacts. This is especially noticeable for Scene-101 (top and bottom rows), where the mesh has large holes where PointNeRF fails to render meaningful pixels, even with their 'growing' heuristic that is aimed towards filling such gaps.