NeRFlex: Resource-aware Real-time High-quality Rendering of Complex Scenes on Mobile Devices

TL;DR

NeRFlex addresses the challenge of real-time, high-quality NeRF rendering for complex scenes on mobile devices by decomposing scenes into multiple sub-scenes, each represented by a lightweight NeRF and optimized under device memory and compute constraints. The system integrates detail-based segmentation to identify high-frequency regions, a white-box profiler to map configurations to baked data size $S$ and quality $Q$, and a dynamic programming-based configuration selector to maximize total rendering quality under a fixed budget $H$. An on-device WebGL pipeline renders the multi-NeRF representations, achieving real-time performance with high fidelity on commodity devices and outperforming single-NeRF and memory-heavy Block-NeRF baselines. The approach demonstrates that resource-aware, multi-NeRF rendering, coupled with principled profiling and optimization, enables practical, high-quality rendering of complex scenes on mobile platforms, with significant implications for AR/VR and mobile graphics.

Abstract

Neural Radiance Fields (NeRF) is a cutting-edge neural network-based technique for novel view synthesis in 3D reconstruction. However, its significant computational demands pose challenges for deployment on mobile devices. While mesh-based NeRF solutions have shown potential in achieving real-time rendering on mobile platforms, they often fail to deliver high-quality reconstructions when rendering practical complex scenes. Additionally, the non-negligible memory overhead caused by pre-computed intermediate results complicates their practical application. To overcome these challenges, we present NeRFlex, a resource-aware, high-resolution, real-time rendering framework for complex scenes on mobile devices. NeRFlex integrates mobile NeRF rendering with multi-NeRF representations that decompose a scene into multiple sub-scenes, each represented by an individual NeRF network. Crucially, NeRFlex considers both memory and computation constraints as first-class citizens and redesigns the reconstruction process accordingly. NeRFlex first designs a detail-oriented segmentation module to identify sub-scenes with high-frequency details. For each NeRF network, a lightweight profiler, built on domain knowledge, is used to accurately map configurations to visual quality and memory usage. Based on these insights and the resource constraints on mobile devices, NeRFlex presents a dynamic programming algorithm to efficiently determine configurations for all NeRF representations, despite the NP-hardness of the original decision problem. Extensive experiments on real-world datasets and mobile devices demonstrate that NeRFlex achieves real-time, high-quality rendering on commercial mobile devices.

Figures (9)

• Figure 1: System overview of NeRFlex, an on-device real-time, high-quality, complex scene rendering system.
• Figure 2: Working pipeline of the segmentation module. $f$ refers to the maximum frequency of the object, while $\alpha$ refers to the threshold.
• Figure 3: Emperical evaluation of our profiling models
• Figure 4: Rendering results of a complex scene with SSIM scores for high-frequency detail region on iPhone 13 (memory constraint: 240MB): NeRFlex outperforms baselines in visual quality and meeting the memory constraint. Single-NeRF model yields poor quality. Block-NeRF achieves the highest rendering quality but is not applicable in mobile settings.
• Figure 5: The overall performance of different approaches on two representative mobile devices across different simulated scenes, the term Single refers to the MobileNeRF