SpectralGaussians: Semantic, spectral 3D Gaussian splatting for multi-spectral scene representation, visualization and analysis

TL;DR

This work tackles multi-spectral scene representation and rendering by extending 3D Gaussian Splatting to the spectral domain. It introduces an end-to-end Spectral Gaussian Splatting framework that jointly learns BRDF, lighting, per-spectrum features, and semantic segmentation from multi-view spectrum maps using a differentiable rasterizer. Key contributions include a per-spectrum physically-based shading model, spectral semantic representations with Identity Encoding, and spectral scene editing capabilities. Empirical results show the proposed approach outperforms prior spectral NeRF variants (e.g., XNeRF, SpectralNeRF) and non-spectral Gaussian splatting on synthetic and real datasets, while enabling efficient rendering and spectral analysis for editing and analysis.

Abstract

We propose a novel cross-spectral rendering framework based on 3D Gaussian Splatting (3DGS) that generates realistic and semantically meaningful splats from registered multi-view spectrum and segmentation maps. This extension enhances the representation of scenes with multiple spectra, providing insights into the underlying materials and segmentation. We introduce an improved physically-based rendering approach for Gaussian splats, estimating reflectance and lights per spectra, thereby enhancing accuracy and realism. In a comprehensive quantitative and qualitative evaluation, we demonstrate the superior performance of our approach with respect to other recent learning-based spectral scene representation approaches (i.e., XNeRF and SpectralNeRF) as well as other non-spectral state-of-the-art learning-based approaches. Our work also demonstrates the potential of spectral scene understanding for precise scene editing techniques like style transfer, inpainting, and removal. Thereby, our contributions address challenges in multi-spectral scene representation, rendering, and editing, offering new possibilities for diverse applications.

Figures (7)

• Figure 1: The proposed spectral Gaussian splatting framework: Spectral Gaussian model predicting BRDF parameters, distilled feature fields, and light per spectrum from multi-view spectrum-maps. The full-spectra maps and learnable parameters are introduced later in the training process by initializing them with priors from all other spectra.
• Figure 2: Spectral scene editing: The segmented scene at 450nm (middle) is used to perform a semantic style-transfer on the full spectra (left). The semantic stylized scene (right) has been generated using by applying a style transfer on the multi-view maps (full-spectra) and then in-painting the splats using the semantic object-ID in spectrum 450nm.
• Figure 3: Snapshot of the different scenes in the Spectral NeRF synthetic and Spectral shiny Blender datasets
• Figure 4: Qualitative comparison of CrossSpectralNerf poggi2022xnerf with our method with the dino dataset.
• Figure 5: Qualitative comparison of CrossSpectralNerf poggi2022xnerf with our method with the Penguin dataset. The comparison shows the average of the 10 spectra colored with colormap viridis (left) and one such spectra (right)