GSsplat: Generalizable Semantic Gaussian Splatting for Novel-view Synthesis in 3D Scenes

TL;DR

GSsplat tackles the challenge of generalizable semantic view synthesis in unseen 3D scenes by predicting scene-adaptive color and semantic Gaussians from multi-view inputs, without scene-specific training. It combines a universal hybrid network for color and semantic features with a point-level 3D space interaction and a Gaussian offset learning module that uses group-based supervision to adapt Gaussian positions. The approach yields state-of-the-art semantic synthesis results while offering the fastest feed-forward speed among generalizable multi-task methods, albeit with color rendering nuances due to the explicit 3D Gaussian representation. Overall, GSsplat enables efficient, high-quality semantic reconstruction across novel views and has strong potential for real-time 3D scene understanding applications.

Abstract

The semantic synthesis of unseen scenes from multiple viewpoints is crucial for research in 3D scene understanding. Current methods are capable of rendering novel-view images and semantic maps by reconstructing generalizable Neural Radiance Fields. However, they often suffer from limitations in speed and segmentation performance. We propose a generalizable semantic Gaussian Splatting method (GSsplat) for efficient novel-view synthesis. Our model predicts the positions and attributes of scene-adaptive Gaussian distributions from once input, replacing the densification and pruning processes of traditional scene-specific Gaussian Splatting. In the multi-task framework, a hybrid network is designed to extract color and semantic information and predict Gaussian parameters. To augment the spatial perception of Gaussians for high-quality rendering, we put forward a novel offset learning module through group-based supervision and a point-level interaction module with spatial unit aggregation. When evaluated with varying numbers of multi-view inputs, GSsplat achieves state-of-the-art performance for semantic synthesis at the fastest speed.

Figures (8)

• Figure 1: The top is the non-generalizable 3DGS reconstruction process and the bottom is the generalizable inference.
• Figure 2: Overview of GSsplat. Given $N$ source views and camera poses, our model directly predicts the semantic Gaussian and color Gaussian parameters from RGB and depth information for 3D scene reconstruction. Firstly, the hybrid network uses a multi-view encoding module to extract 2D semantic and color features. Next, the features are decoded to the original image resolution and unprojected per pixel to the 3D space. After point-level interaction and Gaussian mapping, the semantic and color Gaussian radiance fields are constructed by the predicted parameters, and novel views are rendered through the Gaussian splatting operation.
• Figure 3: Point-level interaction module. The left is the cross-view interaction method based on 2D features in other multi-view reconstruction approaches, and the right presents the process of our point-level interaction through space unit aggregation.
• Figure 4: Offset learning module. We divide the initial Gaussian centers (red) into two groups, one with offset (green) and one without offset (blue), and supervise them from both the rendering and geometric projection directions.
• Figure 5: The demonstration of the novel-view synthesis. Our method is compared with GSNeRF chou2024gsnerf on 8-view inputs on evaluation datasets. The first two rows are from the test data of ScanNet, and the last two rows are from Replica.