CSGaussian: Progressive Rate-Distortion Compression and Segmentation for 3D Gaussian Splatting

TL;DR

This work tackles the high transmission cost and limited decode-time segmentation of 3D Gaussian Splatting by introducing CSGaussian, a three-stage, rate-distortion-optimized compression and segmentation framework. It combines an INR-based hyperprior with a progressive training scheme that first optimizes color, then learns and compresses semantic features, and finally performs RD-optimized semantic compression. Key contributions include the INR-based hyperprior for color and semantic attributes, and compression-guided segmentation with quantization-aware training and quality-aware weighting to balance rendering fidelity and semantic accuracy. Experiments on LERF and 3D-OVS show substantial bitrate reductions while preserving rendering quality and improving open-vocabulary segmentation, enabling efficient server-to-edge applications such as AR scene editing and manipulation.

Abstract

We present the first unified framework for rate-distortion-optimized compression and segmentation of 3D Gaussian Splatting (3DGS). While 3DGS has proven effective for both real-time rendering and semantic scene understanding, prior works have largely treated these tasks independently, leaving their joint consideration unexplored. Inspired by recent advances in rate-distortion-optimized 3DGS compression, this work integrates semantic learning into the compression pipeline to support decoder-side applications--such as scene editing and manipulation--that extend beyond traditional scene reconstruction and view synthesis. Our scheme features a lightweight implicit neural representation-based hyperprior, enabling efficient entropy coding of both color and semantic attributes while avoiding costly grid-based hyperprior as seen in many prior works. To facilitate compression and segmentation, we further develop compression-guided segmentation learning, consisting of quantization-aware training to enhance feature separability and a quality-aware weighting mechanism to suppress unreliable Gaussian primitives. Extensive experiments on the LERF and 3D-OVS datasets demonstrate that our approach significantly reduces transmission cost while preserving high rendering quality and strong segmentation performance.

Figures (7)

• Figure 1: Comparison of approaches for efficient 3DGS segmentation at the receiver. Left: The naive solution of learning semantics at the receiver is impractical, as it either (a) requires transmitting all training-view images, or (b) relies on rendered views that yield poor-quality SAM masks and suboptimal segmentation. Right: Our proposed method introduces sender-side compression-guided segmentation learning and transmits the semantically enriched 3DGS with RD-optimized compression, enabling efficient 3D segmentation at the receiver.
• Figure 2: Visualization of training view, SAM mask, and 3DGS renderings with and without compression. Background regions of 3DGS, especially when compressed, contain low-quality primitives that fail to represent objects meaningfully. This discrepancy with SAM masks hinders segmentation learning when all primitives are treated equally.
• Figure 3: Overview of our proposed framework. Top (a): The left illustrates the overall architecture of our compression system with an INR-based hyperprior, and the right details attribute coding for attributes $\boldsymbol{a}$ and channel-wise autoregressive modeling (CARM). For better visualization, only one INR module is depicted, though two models are employed to compress color and semantic features. Bottom (b): Pipeline of compression-guide segmentation learning, including quantization-aware training and quality-aware weighting. The weighting visualization (bottom right) shows foreground anchors are emphasized with larger weightings while background ones are down-weighted.
• Figure 4: Performance comparison, where the x axis represents bitrate, and y axis represents rendering quality or segmentation accuracy.
• Figure 5: Rate-mIoU performance comparison for each ablation study.