ThermoSplat: Cross-Modal 3D Gaussian Splatting with Feature Modulation and Geometry Decoupling

TL;DR

ThermoSplat tackles robust cross-modal 3D scene reconstruction from RGB and thermal data by extending 3D Gaussian Splatting with spectral-aware mechanisms. It introduces Cross-Modal FiLM Modulation to condition shared latent features on thermal structural priors, and Modality-Adaptive Geometric Decoupling to allow independent infrared geometry; a hybrid rendering pipeline combines explicit SH with implicit decoding to preserve details. On the RGBT-Scenes dataset, ThermoSplat achieves state-of-the-art rendering quality in both RGB and thermal views and demonstrates robust cross-modal consistency. These innovations offer practical benefits for all-weather perception and multi-spectral robotics applications.

Abstract

Multi-modal scene reconstruction integrating RGB and thermal infrared data is essential for robust environmental perception across diverse lighting and weather conditions. However, extending 3D Gaussian Splatting (3DGS) to multi-spectral scenarios remains challenging. Current approaches often struggle to fully leverage the complementary information of multi-modal data, typically relying on mechanisms that either tend to neglect cross-modal correlations or leverage shared representations that fail to adaptively handle the complex structural correlations and physical discrepancies between spectrums. To address these limitations, we propose ThermoSplat, a novel framework that enables deep spectral-aware reconstruction through active feature modulation and adaptive geometry decoupling. First, we introduce a Cross-Modal FiLM Modulation mechanism that dynamically conditions shared latent features on thermal structural priors, effectively guiding visible texture synthesis with reliable cross-modal geometric cues. Second, to accommodate modality-specific geometric inconsistencies, we propose a Modality-Adaptive Geometric Decoupling scheme that learns independent opacity offsets and executes an independent rasterization pass for the thermal branch. Additionally, a hybrid rendering pipeline is employed to integrate explicit Spherical Harmonics with implicit neural decoding, ensuring both semantic consistency and high-frequency detail preservation. Extensive experiments on the RGBT-Scenes dataset demonstrate that ThermoSplat achieves state-of-the-art rendering quality across both visible and thermal spectrums.

Figures (4)

• Figure 1: Overview of the proposed ThermoSplat framework. Given multi-spectral inputs, our method optimizes 3D Gaussian primitives with decoupled properties. (a) Cross-Modal FiLM Modulation dynamically conditions shared latent features on thermal structural priors to guide visible texture synthesis. (b) Modality-Adaptive Geometric Decoupling resolves geometric inconsistencies between visible and infrared spectrums. (c) The Hybrid Rendering pipeline integrates explicit Spherical Harmonics (SH) with implicit neural decoding, ensuring high-frequency detail preservation and cross-modal semantic consistency.
• Figure 2: Thermal rendering results with and without geometric decoupling. The thermal rendering results without geometric decoupling may inherit sharp textures and high-frequency noise from the visible spectrum.
• Figure 3: Feature level reconstruction loss on the rasterized feature maps. Left: rendered feature map, right: reconstructed RGB-thermal scene. Note that $\mathcal{A}_{f}$ and $\mathcal{A}_{f(t)}$ are only different in the opacity used in rasterization.
• Figure 4: Qualitative comparison of novel view synthesis results on the RGBT-Scenes dataset. We compare ThermoSplat against state-of-the-art multi-spectral reconstruction methods ThermalGaussian thermalgaussian, MS-Splattingv2 mssplattingvmv, MMOne mmone, and the 3DGS baseline 3dgs. Our method generates more accurate rendering results and structural details.