S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

Abstract

Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often repeatedly perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length and limiting scalability. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal, incremental 3D Gaussian semantic field framework: it does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames, enabling scalable online joint reconstruction and understanding. S2GS adopts a geometry-semantic decoupled dual-backbone design: the geometry branch performs causal modeling to drive incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict segmentation masks and identity embeddings, further stabilized by query-level contrastive alignment and lightweight online association with an instance memory. Experiments show that S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks, while significantly improving long-horizon scalability: it processes 1,000+ frames with much slower growth in runtime and GPU memory, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting.

Figures (7)

• Figure 1: Comparison of current-frame inference time and GPU memory usage between S2GS (Ours) and the recent advanced joint reconstruction and understanding method, SIU3Rsiu3r, under varying sequence lengths in the online setting.
• Figure 2: Overview of S2GS. S2GS processes an uncalibrated and unposed RGB image stream in a strictly causal manner. A causal Transformer encoder, guided by geometric priors from a 3D foundation model, predicts camera parameters, depth, and Gaussian attributes to incrementally construct 3D Gaussian representation. A decoupled semantic stream leverages a 2D foundation model and a query-driven decoder to produce per-view semantic and instance predictions. Query-level contrastive learning and an online instance memory bank (MB) stabilize instance identities over time. Semantic confidence is lifted to the 3D Gaussian field and decoded via splatting, enabling unified novel view synthesis, semantic segmentation, instance segmentation, and panoptic segmentation without revisiting past frames.
• Figure 3: Qualitative results on ScanNet dataset.
• Figure 4: Late-stage novel view synthesis results under longer input streams.
• Figure 5: Open-vocabulary language grounding with text queries.