Lightplane: Highly-Scalable Components for Neural 3D Fields

TL;DR

Lightplane introduces LightplaneRenderer and Splatter to dramatically reduce memory usage in 2D-3D mapping for neural 3D fields, enabling processing of many high-resolution images with modest compute. Built on a hybrid hashed representation (voxel grids and triplanes) and ray-wise kernel fusion with selective recomputation, it achieves up to four orders of magnitude memory savings while maintaining speed. The framework supports high-resolution image-level supervision, large-scale multi-view reconstruction, and diffusion-based 3D generation on CO3Dv2, improving depth accuracy, geometry, and generation fidelity. The approach scales to 100+ input views, broadening practical applicability and offering a versatile plugin for diverse 3D tasks; the authors provide open-source code to facilitate adoption.

Abstract

Contemporary 3D research, particularly in reconstruction and generation, heavily relies on 2D images for inputs or supervision. However, current designs for these 2D-3D mapping are memory-intensive, posing a significant bottleneck for existing methods and hindering new applications. In response, we propose a pair of highly scalable components for 3D neural fields: Lightplane Render and Splatter, which significantly reduce memory usage in 2D-3D mapping. These innovations enable the processing of vastly more and higher resolution images with small memory and computational costs. We demonstrate their utility in various applications, from benefiting single-scene optimization with image-level losses to realizing a versatile pipeline for dramatically scaling 3D reconstruction and generation. Code: \url{https://github.com/facebookresearch/lightplane}.

Figures (18)

• Figure 1: We introduce the LightplaneRenderer and Splatter, a pair of highly-scalable components for neural 3D fields (left). They address the key memory bottleneck of 2D-3D mapping (i.e. rendering and lifting), and reduce memory usage by up to four orders of magnitude, which dramatically increases the number of images that can be processed. We showcase how they can boost various 3D applications (right).
• Figure 2: Memory usage of our LightplaneRenderer vs. a standard autograd NeRF renderer.
• Figure 3: LightplaneSplatter. (a) On a hash grid with vertex features ${\boldsymbol{v}_i}$: sampling obtains point features $\boldsymbol{v}_{p}$ by interpolating vertex features weighted by inverse distance; splatting updates vertex features by accumulating point feature to vertex using the same weights. (b) Splatter involves three steps. For each 3D point along the ray, Splatter samples its features from prior 3D hash $\hat{\theta}$ (1), calculates features to be splatted using MLP (2), and splats them to zero-initialized $\theta$ (3).
• Figure 4: Lightplane memory & speed benchmark showing the forward (FW and backward (BW) passes of LightplaneRenderer (left) and Splatter (right), compared to the Autograd renderer and lifter from yu2021pixelnerfchan2023genvs. Lightplane exhibits up to 4 orders of magnitude lower memory consumption at comparable speed. All axes are log-scaled.
• Figure 5: Single-scene optimization with image-level losses. The memory efficiency of Lightplane allows rendering high resolution images in a differentiable way and backpropagating image-level losses. We show pre-optimized 3D scenes (in unseen views) and their stylizations with perceptual losses.