Thermal-NeRF: Neural Radiance Fields from an Infrared Camera

TL;DR

This work introduces Thermal-NeRF, the first method that estimates a volumetric scene representation in the form of a NeRF solely from IR imaging, and showcases unparalleled proficiency in recovering NeRFs in visually degraded scenes where RGB-based methods fall short.

Abstract

In recent years, Neural Radiance Fields (NeRFs) have demonstrated significant potential in encoding highly-detailed 3D geometry and environmental appearance, positioning themselves as a promising alternative to traditional explicit representation for 3D scene reconstruction. However, the predominant reliance on RGB imaging presupposes ideal lighting conditions: a premise frequently unmet in robotic applications plagued by poor lighting or visual obstructions. This limitation overlooks the capabilities of infrared (IR) cameras, which excel in low-light detection and present a robust alternative under such adverse scenarios. To tackle these issues, we introduce Thermal-NeRF, the first method that estimates a volumetric scene representation in the form of a NeRF solely from IR imaging. By leveraging a thermal mapping and structural thermal constraint derived from the thermal characteristics of IR imaging, our method showcasing unparalleled proficiency in recovering NeRFs in visually degraded scenes where RGB-based methods fall short. We conduct extensive experiments to demonstrate that Thermal-NeRF can achieve superior quality compared to existing methods. Furthermore, we contribute a dataset for IR-based NeRF applications, paving the way for future research in IR NeRF reconstruction.

Figures (5)

• Figure 1: Overview of the proposed Thermal-NeRF. Initially, a set of 16-bit IR images $\{I_{16}\}$, undergoes thermal mapping to be transformed into 8-bit images $\{I_8\}$. The method includes a scene contraction step, which compresses the indoor space into a predefined, fixed-size bounding box. Utilizing the camera parameters, ray bundles are generated through the contracted indoor scene. These bundles are then sampled to yield sampling points and directions $\{\mathbf{x}_i, \mathbf{d}_i\}$. The samples are encoded and fed into the MLP $f_{\Theta}$. This step aggregates the output radiance $\mathbf{t}_i$ and densities $\mathbf{\sigma}_i$ to compute the thermal value. A unique structural thermal constraint is proposed to optimize the loss within mini-patches formed by stochastic pixels, see Equation \ref{['eq10']}.
• Figure 2: This illustration highlights that our actual sequences were documented in demanding conditions, including low and fluctuating lighting, as well as smoke. The RGB images, taken by camera RealSense D435, demonstrate the visual outcomes. The frames marked in red represents the same area as captured by both IR and RGB cameras, albeit with differing resolutions and fields of view.
• Figure 3: Qualitative evaluation on self-collected sequences under challenging conditions. To enhance the visual assessment of the effects, IR images are intentionally transformed into pseudo-color representations by employing the jet colormap array for the color conversion process. Model: original NeRFmildenhall2021nerf, Mip-NeRF 360barron2022mip, DVGOsun2022improved, and Thermal-NeRF. Specifically, we present results of Thermal NeRF without pose refinement to underscore the significance of pose optimization in achieving optimal outcomes.
• Figure 4: Examples of mesh reconstruction of heat source objects in a scene. Explicit meshes of cup and kettle exported by Thermal-NeRF and other models are shown respectively.
• Figure 5: Ablation qualitative example. Here we show renderings from different Thermal-NeRF ablation variants. The full model produces the best results. We zoom in on crops to highlight differences in the rendered images.