Region-Adaptive Learned Hierarchical Encoding for 3D Gaussian Splatting Data

TL;DR

This paper tackles the challenge of deploying 3D Gaussian Splatting (3DGS) for bandwidth-constrained scenarios by introducing Region-Adaptive Learned Hierarchical Encoding (RALHE). RALHE learns per-attribute, multi-resolution latent representations on an octree-based geometry, paired with a lightweight decoder and an autoregressive entropy model that uses Morton-order contexts to estimate bitrate under a global rate constraint. The framework jointly optimizes 17 attribute-specific latents, decoders, and entropy coders, enabling high rate–distortion performance with low decoding complexity. Experimental results on synthetic-NeRF 3DGS models show PSNR gains up to about 2 dB at sub-1 MB bitrates compared to state-of-the-art baselines, highlighting practical impact for volumetric streaming and immersive applications.

Abstract

We introduce Region-Adaptive Learned Hierarchical Encoding (RALHE) for 3D Gaussian Splatting (3DGS) data. While 3DGS has recently become popular for novel view synthesis, the size of trained models limits its deployment in bandwidth-constrained applications such as volumetric media streaming. To address this, we propose a learned hierarchical latent representation that builds upon the principles of "overfitted" learned image compression (e.g., Cool-Chic and C3) to efficiently encode 3DGS attributes. Unlike images, 3DGS data have irregular spatial distributions of Gaussians (geometry) and consist of multiple attributes (signals) defined on the irregular geometry. Our codec is designed to account for these differences between images and 3DGS. Specifically, we leverage the octree structure of the voxelized 3DGS geometry to obtain a hierarchical multi-resolution representation. Our approach overfits latents to each Gaussian attribute under a global rate constraint. These latents are decoded independently through a lightweight decoder network. To estimate the bitrate during training, we employ an autoregressive probability model that leverages octree-derived contexts from the 3D point structure. The multi-resolution latents, decoder, and autoregressive entropy coding networks are jointly optimized for each Gaussian attribute. Experiments demonstrate that the proposed RALHE compression framework achieves a rendering PSNR gain of up to 2dB at low bitrates (less than 1 MB) compared to the baseline 3DGS compression methods.

Figures (5)

• Figure 1: Overview of the proposed 3DGS compression framework. We first voxelize the Gaussian mean positions ($\hbox{\boldmath$\mu$}_{i} \in \mathbb{R}^{3}$) and construct an octree to provide a hierarchical representation of the geometry. We encode the voxelized mean positions ($\tilde{\hbox{\boldmath$\mu$}}_{i} \in \mathbb{R}^{3}$) using GPCC in lossless mode and covariances ($\Tilde{\hbox{\boldmath$\Sigma$}}_{i} \in \mathbb{R}^{3\times 3}$) using vector quantization. We encode the attribute data— opacity and spherical harmonic coefficient— using RALHE, where we jointly train latents, decoder networks, and entropy models for each 3DGS attribute.
• Figure 2: (left) hierarchical representation of 3D geometry using octree and corresponding latents, (right) reconstructing the attributes from quantized latents at different resolutions. The quantized latents are upsampled and fed into the decoder.
• Figure 3: Autoregressive probability model for entropy coding
• Figure 4: Visualization of learned latents for color attributes i.e., $0^{\text{th}}$ order SH: ${\bf C}_{i}^{(0)}$.
• Figure 5: RD cuve comparison with state-of-the-art 3DGS compression methods.