BalanceGS: Algorithm-System Co-design for Efficient 3D Gaussian Splatting Training on GPU

TL;DR

This work tackles three bottlenecks in end-to-end 3D Gaussian Splatting training: skewed density in densification, imbalanced projection workload, and fragmented color memory access. It introduces BalanceGS, an algorithm-system co-design with (i) workload-sensitive Gaussian density control, (ii) similarity-based Gaussian sampling and merging, and (iii) reordering-based memory access mapping. The combined approach yields a 1.44× speedup in end-to-end training on an NVIDIA A100 with negligible degradation in synthesis quality, validated across standard 3DGS benchmarks. The results show that jointly optimizing geometry, computation, and memory is essential for practical, high-quality, real-time 3D reconstruction pipelines and can be extended to related GPU-accelerated rendering tasks.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a promising 3D reconstruction technique. The traditional 3DGS training pipeline follows three sequential steps: Gaussian densification, Gaussian projection, and color splatting. Despite its promising reconstruction quality, this conventional approach suffers from three critical inefficiencies: (1) Skewed density allocation during Gaussian densification, (2) Imbalanced computation workload during Gaussian projection and (3) Fragmented memory access during color splatting. To tackle the above challenges, we introduce BalanceGS, the algorithm-system co-design for efficient training in 3DGS. (1) At the algorithm level, we propose heuristic workload-sensitive Gaussian density control to automatically balance point distributions - removing 80% redundant Gaussians in dense regions while filling gaps in sparse areas. (2) At the system level, we propose Similarity-based Gaussian sampling and merging, which replaces the static one-to-one thread-pixel mapping with adaptive workload distribution - threads now dynamically process variable numbers of Gaussians based on local cluster density. (3) At the mapping level, we propose reordering-based memory access mapping strategy that restructures RGB storage and enables batch loading in shared memory. Extensive experiments demonstrate that compared with 3DGS, our approach achieves a 1.44$\times$ training speedup on a NVIDIA A100 GPU with negligible quality degradation.

Figures (6)

• Figure 1: Example and result comparsion between 3D Gaussian Splatting and BalanceGS
• Figure 2: Overview of BalanceGS: (a) training dataflow of 3DGS. (b) three challenges, and (c) three proposed techniques
• Figure 3: The pipeline of heuristic workload-sensitive density control. We first evaluate Gaussian density and determine the density threshold. Then, we perform adjacent densification and apply blending and rescaling based on the respective density magnitudes.
• Figure 4: Similarity-based Gaussian sampling and merging. When merging, threads with light workload process four pixels. Similarity-based sampling includes color quantization, hash-based grouping and opacity accumulation.
• Figure 5: Evalutation of training speedup on Mip-NeRF360, Tanks&Temples, and Deep Blending datasets.