MrGS: Multi-modal Radiance Fields with 3D Gaussian Splatting for RGB-Thermal Novel View Synthesis

TL;DR

MrGS addresses RGB-T novel view synthesis by extending 3D Gaussian Splatting to a multi-modal radiance field. It introduces a multi-modal appearance embedding with orthogonal extraction to separate RGB and thermal information and uses separate positional embeddings to capture Lambertian vs non-Lambertian reflectance. It also integrates physics-based priors—Fourier heat conduction between Gaussians and depth-aware thermal radiation according to the Stefan-Boltzmann and inverse-square laws—to improve thermal rendering and geometry consistency. Empirical results on ThermoNeRF show high-fidelity RGB-T reconstructions with significantly fewer Gaussians than previous methods, demonstrating both performance and efficiency gains.

Abstract

Recent advances in Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) have achieved considerable performance in RGB scene reconstruction. However, multi-modal rendering that incorporates thermal infrared imagery remains largely underexplored. Existing approaches tend to neglect distinctive thermal characteristics, such as heat conduction and the Lambertian property. In this study, we introduce MrGS, a multi-modal radiance field based on 3DGS that simultaneously reconstructs both RGB and thermal 3D scenes. Specifically, MrGS derives RGB- and thermal-related information from a single appearance feature through orthogonal feature extraction and employs view-dependent or view-independent embedding strategies depending on the degree of Lambertian reflectance exhibited by each modality. Furthermore, we leverage two physics-based principles to effectively model thermal-domain phenomena. First, we integrate Fourier's law of heat conduction prior to alpha blending to model intensity interpolation caused by thermal conduction between neighboring Gaussians. Second, we apply the Stefan-Boltzmann law and the inverse-square law to formulate a depth-aware thermal radiation map that imposes additional geometric constraints on thermal rendering. Experimental results demonstrate that the proposed MrGS achieves high-fidelity RGB-T scene reconstruction while reducing the number of Gaussians.

Figures (6)

• Figure 1: Overview of our MrGS and comparisons of RGB-T scene rendering results with a NeRF-based method.
• Figure 2: Overall framework of the proposed MrGS. MrGS framework simultaneously renders RGB and thermal images from 3D Gaussians. The RGB branch integrates the multi-modal appearance embedding $e_m$ with view-dependent encoding $PE_v$ and SH2RGB conversion for rasterization. The thermal branch extracts the $e_{t}$ via orthogonal extraction, concatenates it with view-invariant encoding $PE_p$, and refines the features via Fourier heat transformation. Thermal image is further guided by a thermal radiation map with $L_{boltz}$.
• Figure 3: Graphical description of the view-invariant component extraction for thermal image with orthogonal decomposition.
• Figure 4: Overall Process for refining thermal feature $F_t$ via Fourier's Heat Transformation.
• Figure 5: Qualitative highlights for RGB-T novel view synthesis.