Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCAR-GS: Spatial Context Attention for Residuals in Progressive Gaussian Splatting

Diego Revilla, Pooja Suresh, Anand Bhojan, Ooi Wei Tsang

TL;DR

SCAR-GS tackles the large storage burden of 3D Gaussian Splatting by introducing a progressive codec that uses Residual Vector Quantization to compress learned features, coupled with a spatially aware autoregressive entropy model guided by a multi-resolution hash grid. A dual-codebook RVQ-VAE and a Rotation Trick for gradient propagation enable an auto-regressive predictor to model $P(k_m \mid k_{<m}, \mathbf{h}_{spatial})$, supporting base and refinement layers for progressive transmission. The method demonstrates competitive rate-distortion with smoother progressive refinement and lower storage than previous approaches (PCGS and GoDe) across multiple benchmarks (NeRF Synthetic, Tanks & Temples, MipNeRF360, BungeeNeRF), while incurring higher base-layer cost. The work highlights the value of feature-level residuals and spatial context for progressive streaming of 3D scene representations.

Abstract

Recent advances in 3D Gaussian Splatting have allowed for real-time, high-fidelity novel view synthesis. Nonetheless, these models have significant storage requirements for large and medium-sized scenes, hindering their deployment over cloud and streaming services. Some of the most recent progressive compression techniques for these models rely on progressive masking and scalar quantization techniques to reduce the bitrate of Gaussian attributes using spatial context models. While effective, scalar quantization may not optimally capture the correlations of high-dimensional feature vectors, which can potentially limit the rate-distortion performance. In this work, we introduce a novel progressive codec for 3D Gaussian Splatting that replaces traditional methods with a more powerful Residual Vector Quantization approach to compress the primitive features. Our key contribution is an auto-regressive entropy model, guided by a multi-resolution hash grid, that accurately predicts the conditional probability of each successive transmitted index, allowing for coarse and refinement layers to be compressed with high efficiency.

SCAR-GS: Spatial Context Attention for Residuals in Progressive Gaussian Splatting

TL;DR

SCAR-GS tackles the large storage burden of 3D Gaussian Splatting by introducing a progressive codec that uses Residual Vector Quantization to compress learned features, coupled with a spatially aware autoregressive entropy model guided by a multi-resolution hash grid. A dual-codebook RVQ-VAE and a Rotation Trick for gradient propagation enable an auto-regressive predictor to model , supporting base and refinement layers for progressive transmission. The method demonstrates competitive rate-distortion with smoother progressive refinement and lower storage than previous approaches (PCGS and GoDe) across multiple benchmarks (NeRF Synthetic, Tanks & Temples, MipNeRF360, BungeeNeRF), while incurring higher base-layer cost. The work highlights the value of feature-level residuals and spatial context for progressive streaming of 3D scene representations.

Abstract

Recent advances in 3D Gaussian Splatting have allowed for real-time, high-fidelity novel view synthesis. Nonetheless, these models have significant storage requirements for large and medium-sized scenes, hindering their deployment over cloud and streaming services. Some of the most recent progressive compression techniques for these models rely on progressive masking and scalar quantization techniques to reduce the bitrate of Gaussian attributes using spatial context models. While effective, scalar quantization may not optimally capture the correlations of high-dimensional feature vectors, which can potentially limit the rate-distortion performance. In this work, we introduce a novel progressive codec for 3D Gaussian Splatting that replaces traditional methods with a more powerful Residual Vector Quantization approach to compress the primitive features. Our key contribution is an auto-regressive entropy model, guided by a multi-resolution hash grid, that accurately predicts the conditional probability of each successive transmitted index, allowing for coarse and refinement layers to be compressed with high efficiency.
Paper Structure (34 sections, 13 equations, 3 figures, 10 tables)

This paper contains 34 sections, 13 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Gaussian Splatting
  4. Encoded Representations
  5. Autoregressive Entropy Modeling
  6. Methodology
  7. Preliminaries
  8. Residual Vector Quantization
  9. The Rotation Trick for gradient propagation
  10. Spatially-Aware Autoregressive Entropy Modeling
  11. Spatial-Query Attention Mechanism
  12. Optimization Objective
  13. Rendering Loss
  14. Entropy Loss
  15. Quantization Loss
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Visual overview of the SCAR-GS pipeline. The system uses a hierarchical neural representation (Left) encoded via dual-codebook Residual Vector Quantization (Bottom). A spatially-aware autoregressive entropy model (Top) predicts indices for arithmetic coding. The decoder reconstructs features progressively to render Gaussians of increasing fidelity (Right).
  • Figure 2: Comparisons of the different progressive layers on the Flower scene from the Mip-NeRF360 dataset mipnerf360.
  • Figure 3: R-D curve of our method over different $\lambda_{ssim}$ (0.1, ..., 0.4) values in the Bycicle scene from the MipNeRF360 dataset. Benchmarked against PCGS and GoDE.