Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding

Jingming He, Chongyi Li, Shiqi Wang, Sam Kwong

TL;DR

The paper tackles the limitation of relying on 2D supervision and gradient-only cues in 3D Gaussian Splatting by introducing a joint semantic-rendering framework. It leverages an anisotropic 3D Gaussian local encoding based on the Laplace–Beltrami operator to capture directional local geometry and refines semantic features accordingly, while also adaptively allocating Gaussians and SH levels using local semantic and shape cues. A cross-scene knowledge transfer module maintains a pattern basis to reuse learned shape patterns across scenes, accelerating convergence and improving robustness. Experiments on Replica, ScanNet, and Deep Blending show improved segmentation mIoU and rendering PSNR with maintained high frame rates, demonstrating practical gains in dense 3D scene understanding and rendering efficiency.

Abstract

Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.

Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding

TL;DR

The paper tackles the limitation of relying on 2D supervision and gradient-only cues in 3D Gaussian Splatting by introducing a joint semantic-rendering framework. It leverages an anisotropic 3D Gaussian local encoding based on the Laplace–Beltrami operator to capture directional local geometry and refines semantic features accordingly, while also adaptively allocating Gaussians and SH levels using local semantic and shape cues. A cross-scene knowledge transfer module maintains a pattern basis to reuse learned shape patterns across scenes, accelerating convergence and improving robustness. Experiments on Replica, ScanNet, and Deep Blending show improved segmentation mIoU and rendering PSNR with maintained high frame rates, demonstrating practical gains in dense 3D scene understanding and rendering efficiency.

Abstract

Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.
Paper Structure (15 sections, 15 equations, 4 figures, 5 tables)

This paper contains 15 sections, 15 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adaptive Strategies for 3D Gaussian Splatting
  4. Semantic Integration in 3D Gaussian Splatting
  5. Methodology
  6. Anisotropic 3D Gaussian Local Encoding with Laplace Beltrami Operators
  7. Anisotropic 3D Gaussian Chebyshev descriptor
  8. Local Encoding and Feature Propagation
  9. Semantic-Aware Gaussian Pruning and Spherical Harmonics (SH) Adjustment
  10. Continuous Learning Module for Cross-Scene Knowledge Transfer
  11. Experiments
  12. Datasets and Experimental Setups
  13. Main Results
  14. Ablation Studies
  15. Conclusion

Figures (4)

  • Figure 1: Schematic of our joint enhancement framework in 3D semantic Gaussian. The gold arrow denotes the local anisotropic encoding pathway, leveraging each Gaussian’s intrinsic properties and shape details for semantic refinement. The green arrow represents Gaussian pruning and densification, guided by semantic and shape cues to dynamically add or remove Gaussians and adjust spherical harmonics (SH) levels. This joint semantic-rendering framework enables a more integrated enhancement of semantic understanding and rendering.
  • Figure 2: Overview of our proposed pipeline. In the anisotropic 3D Gaussian local encoding, we select multiple local regions within a view frustum. Each local Gaussian is processed with the ALBO to form Chebyshev descriptors, which then enter a transformer-based module to aggregate local geometry information. The refined features are propagated and fused back into individual Gaussians, updating their semantic vectors. In the semantic-aware Gaussian pruning and SH adjustment, new Gaussians are introduced in areas with high gradients to enhance coverage, while semantic-aware pruning eliminates redundant Gaussians and adjusts SH levels. Finally, we maintain a pattern basis to conduct cross-scene knowledge transfer. Guided by local feature statistics, scene-specific projection matrices are aligned with this basis. When residuals and shape complexity surpass a threshold, the basis is updated to accommodate new patterns. This basis is stored and reused for future scenes.
  • Figure 3: Visualization of semantic segmentation results. From top to bottom displays the View image, Ground Truth, results from Feature 3DGS zhou2024feature, Semantic Gaussians guo2024semantic, and our results.
  • Figure 4: Visualization of rendering results. From left to right shows the Original Scene Image, the results from Feature 3DGS zhou2024feature, and our results (Under the same number of iterations).