Sketch&Patch++: Efficient Structure-Aware 3D Gaussian Representation

TL;DR

Sketch&Patch++ tackles the storage and streaming challenges of 3D Gaussian Splatting by introducing a semantic split of Gaussians into SketchGS (high-frequency, boundary-defining) and PatchGS (low-frequency, smooth regions) and by applying category-specific encoding. A hierarchical, multi-criteria density-based clustering discovers structure directly from the optimized Gaussian distribution, followed by adaptive polynomial regression for SketchGS and pruning/quantization for PatchGS, all packaged in a cascaded bitstream. The approach yields substantial compression (up to about $175\times$) with preserved visual fidelity and enables layered progressive streaming, demonstrated across diverse indoor/outdoor scenes and three benchmark datasets. This structure-aware framework reduces bandwidth and storage demands while maintaining rendering quality, offering practical benefits for bandwidth-constrained streaming and resource-limited devices.

Abstract

We observe that Gaussians exhibit distinct roles and characteristics analogous to traditional artistic techniques -- like how artists first sketch outlines before filling in broader areas with color, some Gaussians capture high-frequency features such as edges and contours, while others represent broader, smoother regions analogous to brush strokes that add volume and depth. Based on this observation, we propose a hybrid representation that categorizes Gaussians into (i) Sketch Gaussians, which represent high-frequency, boundary-defining features, and (ii) Patch Gaussians, which cover low-frequency, smooth regions. This semantic separation naturally enables layered progressive streaming, where the compact Sketch Gaussians establish the structural skeleton before Patch Gaussians incrementally refine volumetric detail. In this work, we extend our previous method to arbitrary 3D scenes by proposing a novel hierarchical adaptive categorization framework that operates directly on the 3DGS representation. Our approach employs multi-criteria density-based clustering, combined with adaptive quality-driven refinement. This method eliminates dependency on external 3D line primitives while ensuring optimal parametric encoding effectiveness. Our comprehensive evaluation across diverse scenes, including both man-made and natural environments, demonstrates that our method achieves up to 1.74 dB improvement in PSNR, 6.7% in SSIM, and 41.4% in LPIPS at equivalent model sizes compared to uniform pruning baselines. For indoor scenes, our method can maintain visual quality with only 0.5\% of the original model size. This structure-aware representation enables efficient storage, adaptive streaming, and rendering of high-fidelity 3D content across bandwidth-constrained networks and resource-limited devices.

Figures (13)

• Figure 1: We propose Sketch&Patch++, a hybrid Gaussian representation with 3D structure prior, significantly reducing the storage of 3DGS model by an order of magnitude while maintaining the visual quality.
• Figure 2: Impact of densification on 3DGS models. (a) Relaxed ($\tau=0.01$), (b) medium ($\tau=0.002$), and (c) strict ($\tau=0.0002$) densification thresholds produce 10K, 27K, and 589K Gaussians respectively. Top row: rendered images with PSNR values. Middle row: per-pixel reconstruction error maps. Bright regions indicate higher reconstruction error, while darker areas correspond to well-reconstructed regions. Bottom row: Gaussian distributions. Note the correlation between the density of Gaussians and the sharp/smooth areas.
• Figure 3: Renderings of SketchGS and PatchGS in the Room scene. (a) SketchGS (1.16 million Gaussians) exhibits dense edge-aligned structures. (b) PatchGS (0.43 million Gaussians) shows scattered surface coverage. For each category, we show the 3DGS rendering (left) and Gaussian center rendering (right).
• Figure 4: Characteristic of SketchGS and PatchGS in the Room scene: (a) Gaussian count, (b) density, (c) elongation, and (d) spatial volume. Compared with PatchGS (1.16 million Gaussians), SketchGS (0.43 million Gaussians) are $100\times$ denser, $1.5\times$ more elongated, and $27\times$ more compact, confirming their distinct roles as edge-aligned structures versus scattered surface coverage.
• Figure 5: The streaming pipeline. We propose to build an efficient representation by leveraging the 3D structure information, which can be essentially regarded as a "codec" in the context of the streaming system.