HPC: Hierarchical Point-based Latent Representation for Streaming Dynamic Gaussian Splatting Compression

TL;DR

HPC tackles the challenge of streaming dynamic Gaussian Splatting by introducing a hierarchical point-based latent representation that assigns latent points per Gaussian and aggregates context across multiple scales, reducing parameter redundancy and spatial waste. It further introduces a first-of-its-kind approach to compress neural network parameters alongside latent codes, using a temporal reference GOP strategy to encode energy-efficient inter-frame residuals within an end-to-end rate-distortion framework. Empirical results show substantial bitrate reductions (e.g., approximately 67% BD-BR improvement over strong baselines) while preserving high rendering fidelity on standard FVV datasets, demonstrating state-of-the-art compression performance for streaming dynamic Gaussian Splatting. The work provides extensive ablations that validate the contributions of the latent hierarchy, the intra- and cross-scale aggregations, and the neural-network compression with temporal references, offering a practical pathway toward efficient real-time streaming of free-viewpoint video.

Abstract

While dynamic Gaussian Splatting has driven significant advances in free-viewpoint video, maintaining its rendering quality with a small memory footprint for efficient streaming transmission still presents an ongoing challenge. Existing streaming dynamic Gaussian Splatting compression methods typically leverage a latent representation to drive the neural network for predicting Gaussian residuals between frames. Their core latent representations can be categorized into structured grid-based and unstructured point-based paradigms. However, the former incurs significant parameter redundancy by inevitably modeling unoccupied space, while the latter suffers from limited compactness as it fails to exploit local correlations. To relieve these limitations, we propose HPC, a novel streaming dynamic Gaussian Splatting compression framework. It employs a hierarchical point-based latent representation that operates on a per-Gaussian basis to avoid parameter redundancy in unoccupied space. Guided by a tailored aggregation scheme, these latent points achieve high compactness with low spatial redundancy. To improve compression efficiency, we further undertake the first investigation to compress neural networks for streaming dynamic Gaussian Splatting through mining and exploiting the inter-frame correlation of parameters. Combined with latent compression, this forms a fully end-to-end compression framework. Comprehensive experimental evaluations demonstrate that HPC substantially outperforms state-of-the-art methods. It achieves a storage reduction of 67% against its baseline while maintaining high reconstruction fidelity.

Figures (10)

• Figure 1: Pipeline of the proposed HPC framework. The framework begins with the latest decoded Neural Gaussians $\mathcal{NG}_{t-1}$ as a reference. It then constructs a hierarchical latent representation $\{Z^r_t\}_{r=0}^{L-1}$ (here, $L=3$) by progressively down-sampling their positions into $\{X_{t-1}^r\}_{r=0}^{L-1}$ and pairing them with the decoded latent embeddings $\{\hat{E}_{t}^r\}_{r=0}^{L-1}$. After the Inner-scale Latent Aggregation (ILA) and Cross-scale Latent Aggregation (CLA), these latent points are fed into the prediction heads to obtain inter-frame residuals for deformation. In HPC, both the latent embeddings $\{E_{t}^r\}_{r=0}^{L-1}$ and network parameters $P_t$ are compressed for transmission. We denote the reconstructed elements from the decoder with a hat mark.
• Figure 2: Inner-scale Latent Aggregation (ILA). ILA takes a target point as input and locates its k-nearest neighbors. It then predicts aggregated weights from the relative positions, performs a weighted sum of the neighbor embeddings, and finally passes the result through an MLP to produce the output.
• Figure 3: HPC's compression scheme. We incorporate compression for both the latent embeddings and the neural networks.
• Figure 4: Parameter distributions across adjacent frames in different layers.
• Figure 5: Rate-distortion curves on the N3DV and the Meet Room datasets. Methods marked with $\bigcirc$ and $\bigtriangleup$ respectively denote the online and offline optimized methods. Points and curves closer to the top-left corner indicate better performance.