AutoInt: Automatic Integration for Fast Neural Volume Rendering

TL;DR

AutoInt tackles the bottleneck of per-ray integration in neural volume rendering by learning a grad network that represents a signal's derivative and then reassembling it into an integral network that encodes the antiderivative. By exploiting the fundamental theorem of calculus, the approach computes definite integrals with only two network evaluations, dramatically speeding up rendering while incurring a controlled loss in image quality. The method generalizes to other domains (e.g., computed tomography) and is implemented with a piecewise volume rendering model and a fast, shared-parameter Grad/Integral network scheme. The results demonstrate more than a 10x speedup over state-of-the-art NeRF-style rendering with meaningful practical gains for real-time or interactive applications, alongside clear paths for future improvements in grad-network architectures and nonlinearities.

Abstract

Numerical integration is a foundational technique in scientific computing and is at the core of many computer vision applications. Among these applications, neural volume rendering has recently been proposed as a new paradigm for view synthesis, achieving photorealistic image quality. However, a fundamental obstacle to making these methods practical is the extreme computational and memory requirements caused by the required volume integrations along the rendered rays during training and inference. Millions of rays, each requiring hundreds of forward passes through a neural network are needed to approximate those integrations with Monte Carlo sampling. Here, we propose automatic integration, a new framework for learning efficient, closed-form solutions to integrals using coordinate-based neural networks. For training, we instantiate the computational graph corresponding to the derivative of the network. The graph is fitted to the signal to integrate. After optimization, we reassemble the graph to obtain a network that represents the antiderivative. By the fundamental theorem of calculus, this enables the calculation of any definite integral in two evaluations of the network. Applying this approach to neural rendering, we improve a tradeoff between rendering speed and image quality: improving render times by greater than 10 times with a tradeoff of slightly reduced image quality.

Figures (6)

• Figure 1: Automatic integration for neural volume rendering. During training, a grad network is optimized to represent multi-view images. At test time, we instantiate a corresponding integral network to rapidly evaluate per-ray integrals through the volume.
• Figure 2: AutoInt pipeline. After (1) defining an integral network architecture, (2) AutoInt builds the corresponding grad network, which is (3) optimized to represent a function. (4) Definite integrals can then be computed by evaluating the integral network, which shares parameters with its grad network.
• Figure 3: AutoInt for computed tomography. Left: illustration of the parameterization. Center: sinograms computed for integral networks using different activation functions. In all cases, the ground truth (GT) sinogram is subsampled 8$\times$ and the optimized integral network is sampled to interpolate missing measurements. The Swish activation performs best in terms of peak signal-to-noise ratio (PSNR). Right: a 1D scanline of the sinogram shows that Swish interpolates missing data best while sine activations sitzmann2020siren tend to overfit the measurements.
• Figure 4: Volume rendering pipeline. During training, the grad networks representing volume density $\sigma$ and color $\mathbf{c}$ are optimized for a given set of multi-view images (top left). For inference, the grad networks' parameters are reassembled to form the integral networks, which represent antiderivatives that can be efficiently evaluated to calculate ray integrals through the volume (bottom left). A sampling network predicts the locations of piecewise sections used for evaluating the definite integrals (right).
• Figure 5: Ablation studies. A view of the Lego scene is shown with a varying number of intervals ($N=\{2,4,8,16,32,64\}$) without (left half of the images) and with (right half) the sampling network. PSNR is computed on the 200 test set views.