NeRF-XL: Scaling NeRFs with Multiple GPUs

TL;DR

NeRF-XL presents a principled, scalable multi-GPU framework for training and rendering Neural Radiance Fields by partitioning the scene into non-overlapping tiles and jointly training across GPUs with forward-pass synchronization only. It rewrites the volume rendering and distortion loss computations to enable tile-wise, partition-based accumulation, dramatically reducing cross-GPU communication from per-sample to per-tile data and eliminating blending artifacts. By avoiding model capacity redundancy and ensuring consistent per-camera embeddings through joint optimization, NeRF-XL reveals multi-GPU scaling laws for NeRFs, achieving quality and speed gains as hardware is added. The approach is demonstrated on diverse datasets, including MatrixCity (258K images over 25km^2), showing robust improvements in reconstruction quality and rendering efficiency with increasing GPU resources.

Abstract

We present NeRF-XL, a principled method for distributing Neural Radiance Fields (NeRFs) across multiple GPUs, thus enabling the training and rendering of NeRFs with an arbitrarily large capacity. We begin by revisiting existing multi-GPU approaches, which decompose large scenes into multiple independently trained NeRFs, and identify several fundamental issues with these methods that hinder improvements in reconstruction quality as additional computational resources (GPUs) are used in training. NeRF-XL remedies these issues and enables the training and rendering of NeRFs with an arbitrary number of parameters by simply using more hardware. At the core of our method lies a novel distributed training and rendering formulation, which is mathematically equivalent to the classic single-GPU case and minimizes communication between GPUs. By unlocking NeRFs with arbitrarily large parameter counts, our approach is the first to reveal multi-GPU scaling laws for NeRFs, showing improvements in reconstruction quality with larger parameter counts and speed improvements with more GPUs. We demonstrate the effectiveness of NeRF-XL on a wide variety of datasets, including the largest open-source dataset to date, MatrixCity, containing 258K images covering a 25km^2 city area.

Figures (14)

• Figure 1: Our principled multi-GPU distributed training algorithm enables scaling up NeRFs to arbitrarily-large scale.
• Figure 2: Independent Training v.s. Joint Training with multi-GPU. Training multiple NeRFs independently tancik2022blocknerfturki2022megameuleman2023progressively requires each NeRF to model both the focused region and its surroundings, leading to redundancy in model's capacity. In contrast, our joint training approach utilizes non-overlapping NeRFs, thus without any redundancy.
• Figure 3: Independent Training requires Blending for Novel-View Synthesis. Either blending in 2D tancik2022blocknerfmeuleman2023progressively or 3D turki2022mega introduces blurriness into the rendering.
• Figure 4: Independent Training Creates Distinct Camera Optimizations. Camera optimization in NeRF can be achieved by either transforming the inaccurate camera itself or all other cameras along with the underlying 3D scene. Thus, training multiple NeRFs independently with camera optimization may lead to inconsistencies in camera corrections and scene geometry, causing more difficulties for blended rendering.
• Figure 5: Potential Artifacts Caused by 3D Blending. On the left we show Mega-NeRF results trained with 2 GPUs. At $0\%$ overlap, boundary artifacts appear due to independent training, while at $15\%$ overlap, severe artifacts appear due to 3D blending. On the right we illustrate the reason behind this artifact: while each independently trained NeRF renders the correct color, the blended NeRF do not guarantee correct color rendering.