Learning Hierarchical Sparse Transform Coding of 3DGS

TL;DR

This work addresses the memory and latency bottlenecks of 3D Gaussian Splatting (3DGS) by introducing SHTC, an end-to-end sparsity-guided hierarchical transform coding framework. SHTC places a KLT-based base layer to decorrelate anchor features and energy efficiently, followed by a sparsity-aware refinement layer that compresses the KLT residual via learned linear measurements and a sparsity-regularized, unfolded ISTA reconstruction. The approach achieves substantial rate–distortion gains across multiple datasets and reduces decoding latency with a modest parameter overhead, outperforming state-of-the-art anchor-based and post-training transform methods. By shifting the heavy lifting of decorrelation into the transform stage, SHTC enables simpler entropy models and points toward low-complexity neural codecs for images and videos. The work also provides a detailed architectural and training blueprint that can guide future transform-based compression in unstructured 3D representations and beyond, with code to be released publicly.

Abstract

3D Gaussian Splatting (3DGS) supports fast, high quality, novel view synthesis but has a heavy memory footprint, making the compression of its model crucial. Current state-of-the-art (SOTA) 3DGS compression methods adopt an anchor-based architecture that pairs the Scaffold-GS representation with conditional entropy coding. However, these methods forego the analysis-synthesis transform, a vital mechanism in visual data compression. As a result, redundancy remains intact in the signal and its removal is left to the entropy coder, which computationally overburdens the entropy coding module, increasing coding latency. Even with added complexity thorough redundancy removal is a task unsuited to an entropy coder. To fix this critical omission, we introduce a Sparsity-guided Hierarchical Transform Coding (SHTC) method, the first study on the end-to-end learned neural transform coding of 3DGS. SHTC applies KLT to decorrelate intra-anchor attributes, followed by quantization and entropy coding, and then compresses KLT residuals with a low-complexity, scene-adaptive neural transform. Aided by the sparsity prior and deep unfolding technique, the learned transform uses only a few trainable parameters, reducing the memory usage. Overall, SHTC achieves an appreciably improved R-D performance and at the same time higher decoding speed over SOTA. Its prior-guided, parameter-efficient design may also inspire low-complexity neural image and video codecs. Our code will be released at https://github.com/hxu160/SHTC_for_3DGS_compression.

Figures (10)

• Figure 1: Illustration of Scaffold-GS and anchor-based compression methods.
• Figure 2: Comparison of inter-channel correlation (visualized as absolute Pearson correlation matrices) and energy compaction in the anchor features of HAC/HAC++ and their KLT coefficients, using the 'playroom' scene as an example.
• Figure 3: Comparison of our method with existing anchor-based 3DGS compression methods.
• Figure 4: Comparison of our method with several conceptually representative baslines.
• Figure 5: Comparison of our method with anchor-based methods on two additional datasets, Synthetic-NeRF mildenhall2021nerf and BungeeNeRF xiangli2022bungeenerf.