A General Implicit Framework for Fast NeRF Composition and Rendering

TL;DR

This work addresses the challenge of real-time composition and shadowing for multiple NeRF objects. It introduces Neural Depth Fields (NeDF), a 5D implicit surface representation, coupled with an Intersection Network that uses a two-level bin classifier to efficiently determine ray-surface intersections, enabling fast, geometry-aware depth queries without explicit spatial structures. The proposed three-step pipeline—NeDF generation, deferred shading, and dynamic shadow casting—supports arbitrary affine transformations and interactive editing, achieving substantial speedups (roughly 30–40x over vanilla NeRF) and compatibility with existing NeRF models (e.g., Mip-NeRF, SNeRF, Neus) within a CUDA-accelerated framework. While primarily designed for solid surfaces, the approach enables dynamic shadow rendering and progressive scene composition, and can be integrated with traditional rendering pipelines for mixed workflows, offering a practical, scalable tool for rapid NeRF scene previews and editing.

Abstract

A variety of Neural Radiance Fields (NeRF) methods have recently achieved remarkable success in high render speed. However, current accelerating methods are specialized and incompatible with various implicit methods, preventing real-time composition over various types of NeRF works. Because NeRF relies on sampling along rays, it is possible to provide general guidance for acceleration. To that end, we propose a general implicit pipeline for composing NeRF objects quickly. Our method enables the casting of dynamic shadows within or between objects using analytical light sources while allowing multiple NeRF objects to be seamlessly placed and rendered together with any arbitrary rigid transformations. Mainly, our work introduces a new surface representation known as Neural Depth Fields (NeDF) that quickly determines the spatial relationship between objects by allowing direct intersection computation between rays and implicit surfaces. It leverages an intersection neural network to query NeRF for acceleration instead of depending on an explicit spatial structure.Our proposed method is the first to enable both the progressive and interactive composition of NeRF objects. Additionally, it also serves as a previewing plugin for a range of existing NeRF works.

Figures (10)

• Figure 1: An overview of our framework (left) and data flow (right). In terms of framework, we use Neural Depth Fields (NeDF) as the representation of the implicit surface within each object's local space during training, under the supervision of pre-trained NeRF. There are three main steps in render time: 1) Step1 is "NeDFs generation step", depth buffer and ID buffer are created by a collection of NeDFs; 2) Step2 is "Deferred shading step", those 2 buffers are then used by a collection of NeRFs to fill a color buffer with no shadows; 3) Step3 is "Shadow step", shadow rays are generated based on the depth buffer and NeDFs are used again to provide a shadow map. Finally, the color buffer and shadow buffer are multiplied to produce the results.
• Figure 2: (a) Given a ray traveling through an object's local space, the first intersection on the surface can be obtained by the distance between the intersection and the tangency point $\mathbf{p}_{\perp}$ (The sphere is in the center of local space). Since $\mathbf{p}_{\perp}$ can be quickly and uniquely determined, this representation leads to a direct intersection computation. (b) To improve the stability of NeDF, we regularize the main body of each ray, approximated by a tuple of points sampled along the ray, within the relaxed bounding box before fedding it into the intersection network.
• Figure 3: Directly regressing $\mu$ may result in significant errors in the discontinuity area (a) during composition (see the top row of c). We address this problem by replacing the regression of $\mu$ with a multi-level classifier (b), which yields a better result (see the bottom row of c).
• Figure 4: Visual quality illustration of the general plugin. For each method, the left column shows the previewing result, the middle column shows the naive result, and the right column visualizes the flip errors Andersson2020.
• Figure 5: Number of objects contained in b3,b10,b20,b50 scene. The number in two ball cases is empty, which is because we use them for the capability pressure test and repeated basketballs and footballs (sharing fifty-fifty) in the 100-ball and 1k-ball scenes.