Thermal3D-GS: Physics-induced 3D Gaussians for Thermal Infrared Novel-view Synthesis

TL;DR

This work targets thermal infrared novel-view synthesis, where physics such as atmospheric transmission and thermal conduction introduce floaters and edge blur that degrade quality. It proposes Thermal3D-GS, a physics-informed 3D Gaussian splatting framework that separately models atmospheric attenuation via an Atmospheric Transmission Field and edge refinement via a Thermal Conduction Module, complemented by a temperature-consistency loss. A new large-scale thermal IR NVS dataset, TI-NSD, with 20 authentic scenes and 6,664 frames, provides a benchmark for evaluating methods in this domain. Empirical results on TI-NSD show an average PSNR improvement of $3.03$ dB over the baseline 3D-GS and superior visual quality, underscoring the value of incorporating infrared physics into neural rendering for robust, all-weather novel-view synthesis.

Abstract

Novel-view synthesis based on visible light has been extensively studied. In comparison to visible light imaging, thermal infrared imaging offers the advantage of all-weather imaging and strong penetration, providing increased possibilities for reconstruction in nighttime and adverse weather scenarios. However, thermal infrared imaging is influenced by physical characteristics such as atmospheric transmission effects and thermal conduction, hindering the precise reconstruction of intricate details in thermal infrared scenes, manifesting as issues of floaters and indistinct edge features in synthesized images. To address these limitations, this paper introduces a physics-induced 3D Gaussian splatting method named Thermal3D-GS. Thermal3D-GS begins by modeling atmospheric transmission effects and thermal conduction in three-dimensional media using neural networks. Additionally, a temperature consistency constraint is incorporated into the optimization objective to enhance the reconstruction accuracy of thermal infrared images. Furthermore, to validate the effectiveness of our method, the first large-scale benchmark dataset for this field named Thermal Infrared Novel-view Synthesis Dataset (TI-NSD) is created. This dataset comprises 20 authentic thermal infrared video scenes, covering indoor, outdoor, and UAV(Unmanned Aerial Vehicle) scenarios, totaling 6,664 frames of thermal infrared image data. Based on this dataset, this paper experimentally verifies the effectiveness of Thermal3D-GS. The results indicate that our method outperforms the baseline method with a 3.03 dB improvement in PSNR and significantly addresses the issues of floaters and indistinct edge features present in the baseline method. Our dataset and codebase will be released in \href{https://github.com/mzzcdf/Thermal3DGS}{\textcolor{red}{Thermal3DGS}}.

Figures (4)

• Figure 1: The imaging process is influenced by atmospheric transmission and thermal conduction. In the image, high-temperature objects exhibit brightness, while low-temperature objects appear dark. The left side primarily depicts the atmospheric transmission effect, resulting in directly synthesizing the novel-view image will result in floaters. The right side predominantly depicts the thermal conduction effect, leading to directly synthesizing new perspective images will result in blurred edges.
• Figure 2: Overview of the proposed framework.
• Figure 3: We show comparisons of ours to previous methods and the corresponding ground truth images from held-out test views. The scenes are, from the top down: Merge, Apples, Standing, Bridge, Tall Building, Basketball Court, UAV1, UAV2, UAV5. Non-obvious differences in quality highlighted by arrows/insets.
• Figure 4: Visualization of ablation study. On the left, ATF eradicates floaters in the synthesized novel-view image. On the right, TCM effectively enhances blurred edges.