Virtualized 3D Gaussians: Flexible Cluster-based Level-of-Detail System for Real-Time Rendering of Composed Scenes

TL;DR

This work tackles the real-time rendering bottleneck of large-scale 3D Gaussian Splatting by introducing Virtualized 3D Gaussians (V3DG), a Nanite-inspired cluster-based LOD system. It features an offline Build stage that forms hierarchical Gaussian clusters via local splatting, and an online Selection stage that uses screen-space footprints and a user-defined tolerance $\tau$ to render only perceptible clusters. The approach demonstrates substantial speedups (up to about $6.19\times$ acceleration at far distances) while maintaining visual fidelity comparable to vanilla 3DGS, and provides anti-aliasing benefits for composed scenes containing around $0.1$ billion Gaussians per scene. The work also offers a synthetic/real-world dataset of composed scenes to evaluate LOD effectiveness and discusses practical limitations such as increased storage and the need for a streaming memory workflow, pointing to future improvements for production-scale deployment.

Abstract

3D Gaussian Splatting (3DGS) enables the reconstruction of intricate digital 3D assets from multi-view images by leveraging a set of 3D Gaussian primitives for rendering. Its explicit and discrete representation facilitates the seamless composition of complex digital worlds, offering significant advantages over previous neural implicit methods. However, when applied to large-scale compositions, such as crowd-level scenes, it can encompass numerous 3D Gaussians, posing substantial challenges for real-time rendering. To address this, inspired by Unreal Engine 5's Nanite system, we propose Virtualized 3D Gaussians (V3DG), a cluster-based LOD solution that constructs hierarchical 3D Gaussian clusters and dynamically selects only the necessary ones to accelerate rendering speed. Our approach consists of two stages: (1) Offline Build, where hierarchical clusters are generated using a local splatting method to minimize visual differences across granularities, and (2) Online Selection, where footprint evaluation determines perceptible clusters for efficient rasterization during rendering. We curate a dataset of synthetic and real-world scenes, including objects, trees, people, and buildings, each requiring 0.1 billion 3D Gaussians to capture fine details. Experiments show that our solution balances rendering efficiency and visual quality across user-defined tolerances, facilitating downstream interactive applications that compose extensive 3DGS assets for consistent rendering performance.

Figures (20)

• Figure 1: Framework of our virtualized LOD system, featuring the offline build stage to prepare clusters at various levels of detail, and the online selection stage to select clusters at appropriate levels of detail given rendering conditions.
• Figure 2: Clustering on the 3DGS assetoak from RTMV dataset RTMV. (a) 3D Gaussian primitives$G$ visualized as colored points. (b) Clusters$C_0$ in the finest layer. (c) Cluster groups$CG_0$ used for simplification from the finest layer.
• Figure 3: Local splatting method. Simplified 3D Gaussians are optimized locally by distilling the appearance of original 3D Gaussians.
• Figure 4: Footprint of a cluster group. The bounding sphere of a cluster group is projected onto screen space, occupying several pixels as footprint.
• Figure 5: Left: Iterative simplification. Each two adjacent clusters are simplified to create a new cluster, forming the tree structure. Right: Online selection. Given footprint tolerance $\tau$, clusters are selected with appropriate levels of detail separately.