Multi-Spectral Gaussian Splatting with Neural Color Representation

TL;DR

This work presents MS-Splatting -- a multi-spectral 3D Gaussian Splatting framework that is able to generate multi-view consistent novel views from images of multiple, independent cameras with different spectral domains, and demonstrates the effectiveness of this new technique in agricultural applications to render vegetation indices.

Abstract

We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting (3DGS) framework that is able to generate multi-view consistent novel views from images of multiple, independent cameras with different spectral domains. In contrast to previous approaches, our method does not require cross-modal camera calibration and is versatile enough to model a variety of different spectra, including thermal and near-infra red, without any algorithmic changes. Unlike existing 3DGS-based frameworks that treat each modality separately (by optimizing per-channel spherical harmonics) and therefore fail to exploit the underlying spectral and spatial correlations, our method leverages a novel neural color representation that encodes multi-spectral information into a learned, compact, per-splat feature embedding. A shallow multi-layer perceptron (MLP) then decodes this embedding to obtain spectral color values, enabling joint learning of all bands within a unified representation. Our experiments show that this simple yet effective strategy is able to improve multi-spectral rendering quality, while also leading to improved per-spectra rendering quality over state-of-the-art methods. We demonstrate the effectiveness of this new technique in agricultural applications to render vegetation indices, such as normalized difference vegetation index (NDVI).

Figures (19)

• Figure 1: Visualization of different spectral images, which allow for quality enhancement and compression for radiance fields. Near-infra red (NIR) reveals details not visible in RGB (magenta and yellow box) which allows for finer reconstruction. Light reflectance characteristics between bands (RE: red-edge; R: narrow 32nm red band) are shared (red box), which allows for compression.
• Figure 2: Overview of the MS-Splatting pipeline. After initial Structure-from-Motion registration across all channels simultaneously, we initialize a multi-spectral neural Gaussian model shared for all spectral channels. Thereby, colors of all spectral bands are encoded in a per-Gaussian feature vector and decoded with a tiny MLP. During optimization, the shared Gaussian model and per Gaussian features are optimized by drawing a random view and spectral band under a multi-spectral loss formulation.
• Figure 3: Visual comparison on the FRUIT TREES (top) and SINGLE TREE (bottom) scenes with all available spectral-bands. The used ThermalGaussian method in this comparison is our multi-spectral re-implementation.
• Figure 4: Visual comparison of the thermal scenes BUILD-A, PAN and FACE on ThermalGaussian, ThermoNeRF and ours. The used ThermalGaussian method is the original implementation by thermalgaussian. Ours-Smooth uses the smoothness loss proposed by thermalgaussian.
• Figure 5: Relative changes in PSNR and LPIPS for the lightweight MLP and with varying feature‐embedding dimension $d$, reported in percentage ($\%$). The first feature dimension, $d=2^1$, is used as the reference for both plots. All reported changes are relative to this baseline.