Gen-NeRF: Efficient and Generalizable Neural Radiance Fields via Algorithm-Hardware Co-Design

TL;DR

Gen-NeRF is the first to enable real-time generalizable NeRFs, demonstrating a promising NeRF solution for next-generation AR/VR devices and an accelerator micro-architecture dedicated to accelerating the resulting model workloads from the Gen-Ne RF algorithm.

Abstract

Novel view synthesis is an essential functionality for enabling immersive experiences in various Augmented- and Virtual-Reality (AR/VR) applications, for which generalizable Neural Radiance Fields (NeRFs) have gained increasing popularity thanks to their cross-scene generalization capability. Despite their promise, the real-device deployment of generalizable NeRFs is bottlenecked by their prohibitive complexity due to the required massive memory accesses to acquire scene features, causing their ray marching process to be memory-bounded. To this end, we propose Gen-NeRF, an algorithm-hardware co-design framework dedicated to generalizable NeRF acceleration, which for the first time enables real-time generalizable NeRFs. On the algorithm side, Gen-NeRF integrates a coarse-then-focus sampling strategy, leveraging the fact that different regions of a 3D scene contribute differently to the rendered pixel, to enable sparse yet effective sampling. On the hardware side, Gen-NeRF highlights an accelerator micro-architecture to maximize the data reuse opportunities among different rays by making use of their epipolar geometric relationship. Furthermore, our Gen-NeRF accelerator features a customized dataflow to enhance data locality during point-to-hardware mapping and an optimized scene feature storage strategy to minimize memory bank conflicts. Extensive experiments validate the effectiveness of our proposed Gen-NeRF framework in enabling real-time and generalizable novel view synthesis.

Figures (12)

• Figure 1: Visualizing the typical execution pipeline of generalizable NeRFs wang2021ibrnetreizenstein2021commonwang2022attention, which condition NeRF on source views and enhance density estimation via a ray transformer. This illustration is modified from the visualization style of wang2021ibrnet.
• Figure 2: Profile our generalizable NeRF model on two GPU devices across three datasets with different resolutions.
• Figure 3: Illustrating (left) the sampling process in vanilla NeRFs and (right) our coarse-then-focus sampling strategy.
• Figure 4: Visualize the epipolar geometric relationship among (a) the sampled 3D points and their projections on the source view for one pixel/ray, (b) those for multiple pixels/rays, and (c) the rays with corresponding pixels located on the same line that passes through the epipole $e_n$. These geometric relationships are deductions of the epipolar geometric analysis in hartley2003multiple.
• Figure 5: Visualize our greedy 3D-point-patch partition that divides sampled 3D points into patches to maximize data reuses.