Towards Real-Time Neural Volumetric Rendering on Mobile Devices: A Measurement Study

TL;DR

This paper defines the entire working pipeline of the NeRF serving system and identifies possible control knobs that are critical to the system from the communication, computation, and visual performance perspective, and reveals the effects of these control knobs on system performance.

Abstract

Neural Radiance Fields (NeRF) is an emerging technique to synthesize 3D objects from 2D images with a wide range of potential applications. However, rendering existing NeRF models is extremely computation intensive, making it challenging to support real-time interaction on mobile devices. In this paper, we take the first initiative to examine the state-of-the-art real-time NeRF rendering technique from a system perspective. We first define the entire working pipeline of the NeRF serving system. We then identify possible control knobs that are critical to the system from the communication, computation, and visual performance perspective. Furthermore, an extensive measurement study is conducted to reveal the effects of these control knobs on system performance. Our measurement results reveal that different control knobs contribute differently towards improving the system performance, with the mesh granularity being the most effective knob and the quantization being the least effective knob. In addition, diverse hardware device settings and network conditions have to be considered to fully unleash the benefit of operating under the appropriate knobs

Figures (6)

• Figure 1: End-to-end pipeline of baked-NeRF serving.
• Figure 2: File size and rendering speed under different mesh granularity
• Figure 3: Visual quality, PSNR and SSIM, under different mesh granularity
• Figure 4: File sizes and rendering speed under different texture image patch size and mesh granularity level
• Figure 5: Visual quality under different texture image patch size with granularity 80 and 128