SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understanding

TL;DR

This work proposes SemGS, a feed-forward framework for reconstructing generalizable semantic fields from sparse image inputs that achieves state-of-the-art performance on benchmark datasets, while providing rapid inference and strong generalization capabilities across diverse synthetic and real-world scenarios.

Abstract

Semantic understanding of 3D scenes is essential for robots to operate effectively and safely in complex environments. Existing methods for semantic scene reconstruction and semantic-aware novel view synthesis often rely on dense multi-view inputs and require scene-specific optimization, limiting their practicality and scalability in real-world applications. To address these challenges, we propose SemGS, a feed-forward framework for reconstructing generalizable semantic fields from sparse image inputs. SemGS uses a dual-branch architecture to extract color and semantic features, where the two branches share shallow CNN layers, allowing semantic reasoning to leverage textural and structural cues in color appearance. We also incorporate a camera-aware attention mechanism into the feature extractor to explicitly model geometric relationships between camera viewpoints. The extracted features are decoded into dual-Gaussians that share geometric consistency while preserving branch-specific attributes, and further rasterized to synthesize semantic maps under novel viewpoints. Additionally, we introduce a regional smoothness loss to enhance semantic coherence. Experiments show that SemGS achieves state-of-the-art performance on benchmark datasets, while providing rapid inference and strong generalization capabilities across diverse synthetic and real-world scenarios.

Figures (6)

• Figure 1: We propose SemGS, a novel framework for generalizable semantic 3DGS. Given sparse-view images of an unseen scene, SemGS can rapidly infer semantic maps under novel viewpoints in a single feed-forward pass.
• Figure 2: Overall pipeline of our proposed SemGS. Given sparse-view RGB images as inputs, SemGS leverages a dual-branch architecture (Sec. \ref{['sec:dual-branch-feature']}) to extract both color and semantic features. These features are used to regress multi-view depth maps (Sec. \ref{['sec:depth']}) and are subsequently decoded into per-pixel dual-Gaussians (Sec. \ref{['sec:gaussian-prediction']}). The resulting dual-Gaussians share geometric attributes while maintaining branch-specific attributes, enabling efficient rasterization for synthesizing both novel RGB views and semantic maps.
• Figure 3: Swin Transformer with camera-aware attention. Different colors denote attention windows, and the window shifting mechanism enables model to capture cross-window connections. Camera poses are injected into the attention process to enhance the 3D reasoning ability.
• Figure 4: Qualitative comparisons on ScanNet and ScanNet++ datasets. We demonstrate novel-view semantic rendering results of different methods. Compared to existing works, our method produces sharper object boundaries, fewer misclassified regions, and more spatially consistent segmentations.
• Figure 5: Generalization ability on unseen domains. Different models trained on ScanNet are directly evaluated on Replica synthetic scenes (left) and real-world robot-captured scenes (right). Our SemGS generates more accurate and complete semantic maps than prior works.