Block-NeRF: Scalable Large Scene Neural View Synthesis

TL;DR

Block-NeRF addresses the challenge of scaling neural view synthesis to city-scale environments by decomposing large scenes into independently trained Block-NeRFs that are selectively rendered and composited at inference. The approach introduces appearance embeddings, learned pose refinement, exposure conditioning, transient-object masking, and a visibility-augmented merging pipeline to achieve seamless, scalable rendering across blocks. Datasets from San Francisco demonstrate that a grid of 35 Block-NeRFs trained on millions of images can render an entire neighborhood, with ablations validating the value of each component. The work highlights practical implications for large-scale mapping, simulation, and automated city-scale visualization while noting limitations in dynamic content handling and the need for acceleration techniques for real-time use.

Abstract

We present Block-NeRF, a variant of Neural Radiance Fields that can represent large-scale environments. Specifically, we demonstrate that when scaling NeRF to render city-scale scenes spanning multiple blocks, it is vital to decompose the scene into individually trained NeRFs. This decomposition decouples rendering time from scene size, enables rendering to scale to arbitrarily large environments, and allows per-block updates of the environment. We adopt several architectural changes to make NeRF robust to data captured over months under different environmental conditions. We add appearance embeddings, learned pose refinement, and controllable exposure to each individual NeRF, and introduce a procedure for aligning appearance between adjacent NeRFs so that they can be seamlessly combined. We build a grid of Block-NeRFs from 2.8 million images to create the largest neural scene representation to date, capable of rendering an entire neighborhood of San Francisco.

Figures (11)

• Figure 1: Block-NeRF is a method that enables large-scale scene reconstruction by representing the environment using multiple compact NeRFs that each fit into memory. At inference time, Block-NeRF seamlessly combines renderings of the relevant NeRFs for the given area. In this example, we reconstruct the Alamo Square neighborhood in San Francisco using data collected over 3 months. Block-NeRF can update individual blocks of the environment without retraining on the entire scene, as demonstrated by the construction on the right. Video results can be found on the project website http://waymo.com/research/block-nerf.
• Figure 2: The scene is split into multiple Block-NeRFs that are each trained on data within some radius (dotted orange line) of a specific Block-NeRF origin coordinate (orange dot). To render a target view in the scene, the visibility maps are computed for all of the NeRFs within a given radius. Block-NeRFs with low visibility are discarded (bottom Block-NeRF) and the color output is rendered for the remaining blocks. The renderings are then merged based on each block origin's distance to the target view.
• Figure 3: Our model is an extension of the model presented in mip-NeRF barron2021mip. The first MLP $f_\sigma$ predicts the density $\sigma$ for a position $\mathbf{x}$ in space. The network also outputs a feature vector that is concatenated with viewing direction $\mathbf{d}$, the exposure level, and an appearance embedding. These are fed into a second MLP $f_c$ that outputs the color for the point. We additionally train a visibility network $f_v$ to predict whether a point in space was visible in the training views, which is used for culling Block-NeRFs during inference.
• Figure 4: The appearance codes allow the model to represent different lighting and weather conditions.
• Figure 5: Our model is conditioned on exposure, which helps account for exposure changes present in the training data. This allows users to alter the appearance of the output images in a human-interpretable manner during inference.