HashPoint: Accelerated Point Searching and Sampling for Neural Rendering

TL;DR

HashPoint tackles the bottleneck of point searching and sampling in neural rendering by fusing rasterization-based point indexing with ray marching and introducing adaptive primary-surface sampling. It converts 3D search to 2D image-plane hashing, storing per-pixel point lists in a hash table and employing an adaptive radius to retrieve relevant points with complexity $O(n + m q)$. The adaptive sampling then concentrates resources on the primary surface hit by each ray, using a pseudo-UDF-based weighting to select high-importance samples for volume rendering, enabling large speedups with preserved or improved accuracy across diverse datasets. The method is designed to be easily integrated with existing NeRF variants and point-based rendering pipelines, offering substantial practical impact for real-time neural rendering.

Abstract

In this paper, we address the problem of efficient point searching and sampling for volume neural rendering. Within this realm, two typical approaches are employed: rasterization and ray tracing. The rasterization-based methods enable real-time rendering at the cost of increased memory and lower fidelity. In contrast, the ray-tracing-based methods yield superior quality but demand longer rendering time. We solve this problem by our HashPoint method combining these two strategies, leveraging rasterization for efficient point searching and sampling, and ray marching for rendering. Our method optimizes point searching by rasterizing points within the camera's view, organizing them in a hash table, and facilitating rapid searches. Notably, we accelerate the rendering process by adaptive sampling on the primary surface encountered by the ray. Our approach yields substantial speed-up for a range of state-of-the-art ray-tracing-based methods, maintaining equivalent or superior accuracy across synthetic and real test datasets. The code will be available at https://jiahao-ma.github.io/hashpoint/.

Figures (14)

• Figure 1: Efficient point clouds searching for ray tracing. (A) Top row shows traditional point cloud search strategies: Uniform Grid, K-D tree, and Octree visualized in 2D for clarity. (A) Bottom row shows Hashed-Point Searching, our method of rasterizing the point cloud onto an image plane and then reorganizing the point cloud list to optimize queries, resulting in the construction of a hash table. Each key (as pixel index) in the table maps to the start index in the point list and the count of points within that pixel with $\mathcal{O}(1)$ complexity. (B) The final selection depicts the search mechanism: using a magnified search kernel, the neighbor points of a target ray (black arrow) are swiftly identified through the hash table and assessed based on their distance to the ray.
• Figure 2: Illustration of Adaptive Primary Surface Sampling. (A) The diagram depicts the generation of the searching kernel on the image plane. (B) We project adjacent points to the ray as sample point candidates (red crosses). Each candidate's importance, determined by its distance distribution to points within its radius, influences its preservation for final feature aggregation.
• Figure 3: Comparison of different point cloud selection strategies. (A)K-NP Extract: extracting K nearest point features per ray. (B)Depth Map Sampling: sampling based on dense depth maps. (C)Multi-Surface Sampling: uniform sampling over multiple surfaces. (D)Our method: adaptive sampling on the primary surface.
• Figure 4: Results on the NeRF-Synthesis nerf dataset shows that our primary surface sampling (blue points) is more efficient than Point-NeRF's sampling (red points) while preserving accuracy.
• Figure 5: Comparison on Waymo waymo dataset shows how our primary surface sampling (blue points) more accurately samples the car body than the K nearest point selection of NPLF (red points).