ThermalNeRF: Thermal Radiance Fields

TL;DR

The paper addresses the challenge of reconstructing accurate 3D scenes from RGB and LWIR images by introducing Broad-Spectrum Radiance Fields that assign separate absorption densities for each wavelength and render both RGB and thermal channels via a unified volumetric framework $F_\Theta:(x,d)\mapsto( c_{rgb}, c_{therm}, \sigma_{rgb}, \sigma_{therm})$. It couples two Nerfacto-style radiance fields for the visible and thermal spectra through specialized losses ($\mathcal{L}_{rgb}$, $\mathcal{L}_{therm}$, $\mathcal{L}_{\sigma}$, $\mathcal{L}_{cc}$, $\mathcal{L}_{tv}$) and a two-part regularization on densities to enable thermal super-resolution and cross-spectral consistency. A calibration pipeline aligns RGB and LWIR cameras, using a grid target and COLMAP-based pose estimation to obtain reliable cross-spectral geometry, with scale resolved by known distances. The method is demonstrated on nine real-world scenes and a synthetic dataset, showing improved thermal reconstruction over baselines, the ability to reveal objects occluded in either spectrum, and notable thermal super-resolution guided by high-resolution RGB data. These results suggest practical impact for infrastructure inspection, agriculture, and other multispectral imaging tasks where thermal information is crucial but LWIR data alone is insufficient.

Abstract

Thermal imaging has a variety of applications, from agricultural monitoring to building inspection to imaging under poor visibility, such as in low light, fog, and rain. However, reconstructing thermal scenes in 3D presents several challenges due to the comparatively lower resolution and limited features present in long-wave infrared (LWIR) images. To overcome these challenges, we propose a unified framework for scene reconstruction from a set of LWIR and RGB images, using a multispectral radiance field to represent a scene viewed by both visible and infrared cameras, thus leveraging information across both spectra. We calibrate the RGB and infrared cameras with respect to each other, as a preprocessing step using a simple calibration target. We demonstrate our method on real-world sets of RGB and LWIR photographs captured from a handheld thermal camera, showing the effectiveness of our method at scene representation across the visible and infrared spectra. We show that our method is capable of thermal super-resolution, as well as visually removing obstacles to reveal objects that are occluded in either the RGB or thermal channels. Please see https://yvette256.github.io/thermalnerf for video results as well as our code and dataset release.

Figures (17)

• Figure 1: RGB (top) and thermal (bottom) renderings from our reconstruction of a large crane structure, based on images collected by a Skydio drone. Thermal imaging with 3D reconstruction can be used for building and infrastructure inspection, among many other applications ranging from agriculture to search and rescue.
• Figure 2: Results on our pyrex scene, compared against two Nerfacto-based baselines nerfstudio in which the visible and thermal models share a density field {RGBT} or are completely separate {RGB}{T}. The {RGBT} model fails to recover the glass bowl as visually transparent yet thermally opaque, while the camera pose refinement fails for the thermal component of the {RGB}{T} model, leading to ghosting.
• Figure 3: Left: COLMAP successfully recovers camera poses and 3D feature locations from RGB images of our pyrex scene. Right: COLMAP fails on corresponding LWIR images of the same scene, due to insufficient features. Bottom: example input views for RGB and LWIR.
• Figure 4: Results on our engine scene. The engine is warm from recent driving; thermal imaging is used by mechanics to quickly diagnose problems.
• Figure 5: Results on our sheet scene. The plastic sheet is opaque to visible light but transparent to LWIR, revealing the hot kettle behind it.