FastNeRF: High-Fidelity Neural Rendering at 200FPS

TL;DR

FastNeRF addresses the real-time rendering bottleneck of Neural Radiance Fields by factorizing the NeRF function into a position-dependent deep radiance map and a direction-dependent weight network, enabling compact, cacheable grids. The method replaces compute-heavy MLP evaluations with fast, cache lookups and uses hardware-accelerated ray tracing alongside a collision mesh to efficiently render, achieving around 200Hz on high-end GPUs and up to 3000x speedups over NeRF. Key contributions include a principled caching strategy with explicit memory tradeoffs, a practical GPU-oriented implementation, and demonstrations on static and dynamic scenes (including telepresence), illustrating the approach’s broad applicability. This work paves the way for interactive neural rendering at high resolutions by making NeRF-like representations memory-bound rather than compute-bound.

Abstract

Recent work on Neural Radiance Fields (NeRF) showed how neural networks can be used to encode complex 3D environments that can be rendered photorealistically from novel viewpoints. Rendering these images is very computationally demanding and recent improvements are still a long way from enabling interactive rates, even on high-end hardware. Motivated by scenarios on mobile and mixed reality devices, we propose FastNeRF, the first NeRF-based system capable of rendering high fidelity photorealistic images at 200Hz on a high-end consumer GPU. The core of our method is a graphics-inspired factorization that allows for (i) compactly caching a deep radiance map at each position in space, (ii) efficiently querying that map using ray directions to estimate the pixel values in the rendered image. Extensive experiments show that the proposed method is 3000 times faster than the original NeRF algorithm and at least an order of magnitude faster than existing work on accelerating NeRF, while maintaining visual quality and extensibility.

Figures (22)

• Figure 1: Left: NeRF neural network architecture. $(x,y,z)$ denotes the input sample position, $(\theta, \phi)$ denotes the ray direction and $(r,g,b,\sigma)$ are the output color and transparency values. Right: our FastNeRF architecture splits the same task into two neural networks that are amenable to caching. The position-dependent network $\mathcal{F}_{pos}$ outputs a deep radiance map $(\bm{u}, \bm{v}, \bm{w})$ consisting of $D$ components, while the $\mathcal{F}_{dir}$ outputs the weights for those components $(\beta_1,\dots,\beta_{D})$ given a ray direction as input.
• Figure 2: Speed evaluation of our method and prior work NeRFrebain2020derfliu2021neuralneff2021donerf on the Lego scene from the Realistic 360 Synthetic NeRF dataset, rendered at $800\times800$ pixels. For previous methods, when numbers for the Lego scene were not available, we used an optimistic approximation.
• Figure 3: Qualitative comparison of our method vs NeRF on the dataset of NeRF at $800^2$ pixels using 8 components. Small cache refers to our method cached at $256^3$, and large cache at $768^3$. Varying the cache size allows for trading compute and memory for image quality resembling levels of detail (LOD) in traditional computer graphics.
• Figure 4: Qualitative comparison of our method vs NeRF on the dataset of localLightfieldFusionPaper at $504\times378$ pixels using 6 factors. Small cache refers to our method cached at $256^3$, and large cache at $768^3$.
• Figure 5: Face images rendered using FastNeRF combined with a deformation field network park2020deformable. Thanks to the use of FastNeRF, expression-conditioned images can be rendered at 30FPS.