On Scaling Up 3D Gaussian Splatting Training

TL;DR

This work presents Grendel, a distributed system that scales 3D Gaussian Splatting training across multiple GPUs by partitioning Gaussians and pixel regions and using sparse communication. It introduces an automatic hyperparameter scaling rule based on the independent gradients hypothesis to enable batched training with large batch sizes without sacrificing quality. Empirical results on large-scale, high-resolution scenes show substantial throughput gains and reconstruction quality improvements over single-GPU training and rival methods like CityGaussian. Grendel enables high-detail 3D reconstructions at scales previously impractical with 3DGS and is released as open source.

Abstract

3D Gaussian Splatting (3DGS) is increasingly popular for 3D reconstruction due to its superior visual quality and rendering speed. However, 3DGS training currently occurs on a single GPU, limiting its ability to handle high-resolution and large-scale 3D reconstruction tasks due to memory constraints. We introduce Grendel, a distributed system designed to partition 3DGS parameters and parallelize computation across multiple GPUs. As each Gaussian affects a small, dynamic subset of rendered pixels, Grendel employs sparse all-to-all communication to transfer the necessary Gaussians to pixel partitions and performs dynamic load balancing. Unlike existing 3DGS systems that train using one camera view image at a time, Grendel supports batched training with multiple views. We explore various optimization hyperparameter scaling strategies and find that a simple sqrt(batch size) scaling rule is highly effective. Evaluations using large-scale, high-resolution scenes show that Grendel enhances rendering quality by scaling up 3DGS parameters across multiple GPUs. On the Rubble dataset, we achieve a test PSNR of 27.28 by distributing 40.4 million Gaussians across 16 GPUs, compared to a PSNR of 26.28 using 11.2 million Gaussians on a single GPU. Grendel is an open-source project available at: https://github.com/nyu-systems/Grendel-GS

Figures (15)

• Figure 1: Two large-scale, high-resolution scene reconstructions using Grendel, our distributed 3D Gaussian rendering system. Both images are rendered using 16 GPUs. The left and right images are represented using 40 million and 24 million gaussians respectively. Grendel achieves state of the art quality (PSNR) for both scenes.
• Figure 2: (a) Traditional 3DGS training pipeline using a single GPU vs. (b) Our Grendel system that distributes 3D Gaussians across multiple GPUs to alleviate the GPU memory bottleneck. We also partition the computation in the pixel and batch dimensions to for further speedup. Every square represents a $16\times16$ block of pixels.
• Figure 3: Cumulative percents of $Radius_i$ relative to image width
• Figure 4: We present a heatmap of the per-tile imbalance in the number of rendered Gaussians on the Train Dataset tankstemples. Redder tiles indicate more Gaussian splats. Distant, low-detail areas like the sky need fewer Gaussians than detailed foreground regions like the train, highlighting an imbalance in rendering intensity.
• Figure 5: Each GPU only considers Gaussians whose footprints intersect with its assigned pixel render area.