SpNeRF: Memory Efficient Sparse Volumetric Neural Rendering Accelerator for Edge Devices

TL;DR

SpNeRF tackles memory bottlenecks in edge-accelerated neural rendering by exploiting voxel grid sparsity with a hash-map preprocessing step and an online sparse voxel grid decoding flow, avoiding full grid restoration. A dedicated hardware design comprising a Sparse Grid Processing Unit and an MLP Unit pipelines hash lookups, trilinear interpolation, and MLP inference, with FP16 on-chip data and INT8 off-chip data to minimize memory. The approach achieves around 21x memory size reduction while maintaining PSNR comparable to dense baselines, and delivers substantial speedups and energy efficiency improvements over Jetson edge devices and prior accelerators. Overall, SpNeRF demonstrates practical edge-ready neural rendering acceleration by co-designing sparse data processing with specialized hardware.

Abstract

Neural rendering has gained prominence for its high-quality output, which is crucial for AR/VR applications. However, its large voxel grid data size and irregular access patterns challenge real-time processing on edge devices. While previous works have focused on improving data locality, they have not adequately addressed the issue of large voxel grid sizes, which necessitate frequent off-chip memory access and substantial on-chip memory. This paper introduces SpNeRF, a software-hardware co-design solution tailored for sparse volumetric neural rendering. We first identify memory-bound rendering inefficiencies and analyze the inherent sparsity in the voxel grid data of neural rendering. To enhance efficiency, we propose novel preprocessing and online decoding steps, reducing the memory size for voxel grid. The preprocessing step employs hash mapping to support irregular data access while maintaining a minimal memory size. The online decoding step enables efficient on-chip sparse voxel grid processing, incorporating bitmap masking to mitigate PSNR loss caused by hash collisions. To further optimize performance, we design a dedicated hardware architecture supporting our sparse voxel grid processing technique. Experimental results demonstrate that SpNeRF achieves an average 21.07$\times$ reduction in memory size while maintaining comparable PSNR levels. When benchmarked against Jetson XNX, Jetson ONX, RT-NeRF.Edge and NeuRex.Edge, our design achieves speedups of 95.1$\times$, 63.5$\times$, 1.5$\times$ and 10.3$\times$, and improves energy efficiency by 625.6$\times$, 529.1$\times$, 4$\times$, and 4.4$\times$, respectively.

Figures (14)

• Figure 1: Orginal VQRF flow and our SpNeRF flow
• Figure 2: Time distribution
• Figure 3: Voxel grid data sparsity
• Figure 5: An illustration of preprocessing and online decoding flow
• Figure 6: SpNeRF accelerator architecture