MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures

TL;DR

MobileNeRF addresses the slow rendering of NeRFs on consumer hardware by replacing volumetric ray marching with a polygon rasterization pipeline that rasterizes a textured mesh and uses a lightweight neural deferred shader to produce final colors. The method trains in three stages—continuous mesh-based NeRF, binarization of opacities, and texture-baked discretization—resulting in a textured mesh plus a small fragment-shader network that renders at interactive frame rates on mobile and in web browsers. Key contributions include a 10x speedup over SNeRG with similar image quality, reduced memory footprint through texture baking, and broad device/browser compatibility that enables real-time editing and manipulation of scenes. While offering substantial practical impact for mobile and web-based neural rendering, the approach is limited by binary opacity (no semi-transparency), fixed texture resolution, and absence of explicit global illumination, suggesting avenues for future improvements such as partial sorting, mipmaps, and faster training pipelines.

Abstract

Neural Radiance Fields (NeRFs) have demonstrated amazing ability to synthesize images of 3D scenes from novel views. However, they rely upon specialized volumetric rendering algorithms based on ray marching that are mismatched to the capabilities of widely deployed graphics hardware. This paper introduces a new NeRF representation based on textured polygons that can synthesize novel images efficiently with standard rendering pipelines. The NeRF is represented as a set of polygons with textures representing binary opacities and feature vectors. Traditional rendering of the polygons with a z-buffer yields an image with features at every pixel, which are interpreted by a small, view-dependent MLP running in a fragment shader to produce a final pixel color. This approach enables NeRFs to be rendered with the traditional polygon rasterization pipeline, which provides massive pixel-level parallelism, achieving interactive frame rates on a wide range of compute platforms, including mobile phones.

Figures (10)

• Figure 1: Teaser -- We present a NeRF that can run on a variety of common devices at interactive frame rates.
• Figure 2: Overview (rendering) -- We represent the scene as a triangle mesh textured by deep features. We first rasterize the mesh to a deferred rendering buffer. For each visible fragment, we execute a neural deferred shader that converts the feature and view direction to the corresponding output pixel color.
• Figure 3: Overview (train) -- We initialize the mesh as a regular grid, and use MLPs to represent features and opacity for any point on the mesh. For each ray, we compute its intersection points on the mesh, and alpha-composite the colors of those points to obtain the output color. In a later training stage, we enforce binary opacity, and perform super-sampling on features for anti-aliasing.
• Figure 4: Configurations of polygonal meshes -- The meshes employed for the different types of scenes. We sketch the distribution of camera poses in training views.
• Figure 5: Quadrature points -- are obtained by (a) identifying cells that intersect the ray; (b) pruning cells that do not contain geometry; and, (c) computing explicit intersections with the mesh.