FlexNeRFer: A Multi-Dataflow, Adaptive Sparsity-Aware Accelerator for On-Device NeRF Rendering

TL;DR

FlexNeRFer presents a versatile on-device NeRF accelerator that unifies diverse NeRF models through a flexible NoC, a bit-scalable MAC array, and sparsity-aware data compression. By targeting the bottlenecks in GEMM/GEMV computations and neural feature encoding, it achieves substantial speedups and energy efficiency gains over both a modern GPU and a state-of-the-art NeRF accelerator, while supporting multi-dataflow and adaptive sparsity across 4-, 8-, and 16-bit precisions. Key architectural innovations include a hierarchical distribution network (HMF-NoC), a reduction tree with shifter sharing, and online sparsity-format encoding powered by a sparsity-aware format encoder/decoder, plus a NeRF encoding unit with a PEE and an HEE. The results demonstrate significant practical impact for on-device NeRF rendering in AR/VR and autonomous systems, enabling real-time, energy-efficient scene generation across diverse NeRF models.

Abstract

Neural Radiance Fields (NeRF), an AI-driven approach for 3D view reconstruction, has demonstrated impressive performance, sparking active research across fields. As a result, a range of advanced NeRF models has emerged, leading on-device applications to increasingly adopt NeRF for highly realistic scene reconstructions. With the advent of diverse NeRF models, NeRF-based applications leverage a variety of NeRF frameworks, creating the need for hardware capable of efficiently supporting these models. However, GPUs fail to meet the performance, power, and area (PPA) cost demanded by these on-device applications, or are specialized for specific NeRF algorithms, resulting in lower efficiency when applied to other NeRF models. To address this limitation, in this work, we introduce FlexNeRFer, an energy-efficient versatile NeRF accelerator. The key components enabling the enhancement of FlexNeRFer include: i) a flexible network-on-chip (NoC) supporting multi-dataflow and sparsity on precision-scalable MAC array, and ii) efficient data storage using an optimal sparsity format based on the sparsity ratio and precision modes. To evaluate the effectiveness of FlexNeRFer, we performed a layout implementation using 28nm CMOS technology. Our evaluation shows that FlexNeRFer achieves 8.2~243.3x speedup and 24.1~520.3x improvement in energy efficiency over a GPU (i.e., NVIDIA RTX 2080 Ti), while demonstrating 4.2~86.9x speedup and 2.3~47.5x improvement in energy efficiency compared to a state-of-the-art NeRF accelerator (i.e., NeuRex).

Figures (20)

• Figure 1: Rendering latency of seven representative NeRF models, i.e., NeRF vanilar_nerf, KiloNeRF kilonerf, NSVF NSVF, Mip-NeRF mip_nerf, Instant-NGP instant_ngp, IBRNet ibrnet, and TensoRF TensoRF, on the NVIDIA RTX 2080 Ti using the Synthetic-NeRF dataset synthetic_nerf.
• Figure 2: Visual guide to the execution pipeline of NeRF vanilar_nerf.
• Figure 3: Runtime breakdown of seven modern NeRF models, including NeRF vanilar_nerf, KiloNeRF kilonerf, NSVF NSVF, Mip-NeRF mip_nerf, Instant-NGP instant_ngp, IBRNet ibrnet, and TensoRF TensoRF, evaluated on an NVIDIA RTX 2080 Ti using the Synthetic-NeRF dataset synthetic_nerf. In the figure, when encoding is performed using GEMM/GEMV operations, its execution time is included within the GEMM/GEMV runtime.
• Figure 4: MAC utilization of two commercial accelerators, i.e., NVIDIA NVDLA nvdira and Google TPU tpu_v4, across various scenarios. (a) and (b) illustrate MAC utilization when data from the early and late layers of CNNs are mapped to two accelerators, respectively. (c) and (d) present MAC utilization during irregular dense and sparse GEMM operations, which form the backbone of computations in MLPs and Transformers.
• Figure 5: Data mapping of an irregular sparse GEMM operation onto a 4×4 MAC array in a dense manner. 'B', 'M', and 'U' denote broadcast, multicast, and unicast, respectively.