PRoGS: Progressive Rendering of Gaussian Splats

TL;DR

The paper tackles the challenge of rendering 3D Gaussian Splatting (3DGS) content progressively to reduce memory demands and bandwidth while delivering early, perceptually accurate visuals. It introduces a contribution-based ordering that ranks Gaussians by their accumulated impact across training views, enhanced by an octree refinement, frustum-aware prioritization, and object-level rendering, and demonstrates seamless integration with existing compression methods. Across multiple datasets and metrics, the approach yields faster time-to-first-paint and higher quality at lower data budgets than existing web-based baselines, validating its suitability for on-device and streaming scenarios. This work lays the groundwork for bandwidth-efficient, progressive streaming of 3DGS content with practical implications for OTT delivery and interactive visualization scenarios.

Abstract

Over the past year, 3D Gaussian Splatting (3DGS) has received significant attention for its ability to represent 3D scenes in a perceptually accurate manner. However, it can require a substantial amount of storage since each splat's individual data must be stored. While compression techniques offer a potential solution by reducing the memory footprint, they still necessitate retrieving the entire scene before any part of it can be rendered. In this work, we introduce a novel approach for progressively rendering such scenes, aiming to display visible content that closely approximates the final scene as early as possible without loading the entire scene into memory. This approach benefits both on-device rendering applications limited by memory constraints and streaming applications where minimal bandwidth usage is preferred. To achieve this, we approximate the contribution of each Gaussian to the final scene and construct an order of prioritization on their inclusion in the rendering process. Additionally, we demonstrate that our approach can be combined with existing compression methods to progressively render (and stream) 3DGS scenes, optimizing bandwidth usage by focusing on the most important splats within a scene. Overall, our work establishes a foundation for making remotely hosted 3DGS content more quickly accessible to end-users in over-the-top consumption scenarios, with our results showing significant improvements in quality across all metrics compared to existing methods.

Figures (9)

• Figure 1: Progressive Rendering of 3DGS using our approach versus using an existing web-viewer by Antimatter. From left to right, we show 0.2%, 0.5%, 1%, and 10% of the total number of splats, respectively. Our approach results in a faster visualization of a representative version of the scene. At 0.2%, we have loaded in a basic level of the truck while it is completely missing in the alternative approach.
• Figure 2: Our approach works by rendering all training views and returning the top 20 contributing splats per pixel per image. This information is used to create an ordering which can then be used to dynamically select how many splats are used. For progressive rendering/streaming, we can change the number of splats per sent chunk to start rendering as soon as possible without the user needing to wait. With little overhead in total time, we drastically reduce both time and bandwidth until time to First Paint.
• Figure 3: We show the results of different scenes for all three metrics and that our two approaches consistently outperform the previous methods by a significant margin. These scenes are taken from the MipNeRF360 barron2022mipnerf360 indoor and outdoor datasets and the Tanks&Temples Knapitsch2017 dataset. PSNR is measured compared to the original reconstruction and is capped at 100 when the complete scene is loaded in. (PSNR$\uparrow$, SSIM$\uparrow$, LPIPS$\downarrow$)
• Figure 4: A concrete comparison between the three aforementioned methods on the bonsai scene taken from Mip-NeRF360 indoor scenes being rendered at 10% splats compared to the ground truth. Notice how both the bonsai and cloth are being rendered at a higher quality in Ours(b) than in Antimatter(c). Tree.js(d) will almost always be outperformed using standard metrics as the background is completely missing.
• Figure 5: Despite the right image (b) performing better than the left image (a), we can clearly see there are more errors in the foreground when comparing (c) with (d). Due to most images containing mostly containing closer areas, the gains we achieve in the background are not reflected in the average PSNR scores for the complete scene, as was shown in \ref{['fig:metrics_comp_packed']}. Perceptually, our base approach with the octree performs better overall.