Learning Cross-Spectral Point Features with Task-Oriented Training

TL;DR

The paper tackles the challenge of cross-spectral (thermal-visible) navigation for UAVs by learning cross-spectral point features under task-oriented supervision. It introduces a shared feature network whose outputs feed a differentiable, non-learned registration pipeline, with losses applied to matching and homography estimation to bias features toward cross-spectral utility. Key contributions include a detailed comparison of task-based losses, demonstrated improvement in cross-spectral registration on the MultiPoint dataset, and evidence that the learned features generalize to classical pipelines. The work advances practical cross-spectral fusion for UAV navigation, enabling more robust operation in poor visibility environments and providing a foundation for further geometric integration such as differentiable RANSAC.

Abstract

Unmanned aerial vehicles (UAVs) enable operations in remote and hazardous environments, yet the visible-spectrum, camera-based navigation systems often relied upon by UAVs struggle in low-visibility conditions. Thermal cameras, which capture long-wave infrared radiation, are able to function effectively in darkness and smoke, where visible-light cameras fail. This work explores learned cross-spectral (thermal-visible) point features as a means to integrate thermal imagery into established camera-based navigation systems. Existing methods typically train a feature network's detection and description outputs directly, which often focuses training on image regions where thermal and visible-spectrum images exhibit similar appearance. Aiming to more fully utilize the available data, we propose a method to train the feature network on the tasks of matching and registration. We run our feature network on thermal-visible image pairs, then feed the network response into a differentiable registration pipeline. Losses are applied to the matching and registration estimates of this pipeline. Our selected model, trained on the task of matching, achieves a registration error (corner error) below 10 pixels for more than 75% of estimates on the MultiPoint dataset. We further demonstrate that our model can also be used with a classical pipeline for matching and registration.

Figures (4)

• Figure 1: Conceptual rendering of a *uav using an onboard thermal camera to navigate within a visible-spectrum map. Images from MultiPoint dataset achermann_multipoint_2020.
• Figure 2: Training overview flowchart. One image of a thermal-visible image pair is transformed (i.e., augmented) by a random but known homography. The feature network response to the augmented image pair is fed to a registration (i.e., homography estimation) pipeline. Losses are applied to the network output, matches, and homography estimate. Dashed lines denote loss computation steps.
• Figure 3: Alignment visualization for different *ace values (expressed in units of pixels) on images of shape 512 $\times$ 640. Each panel shows a rectified image in green overlaid with its "recovered’’ version in purple.
• Figure 4: Box plots of *ace expressed in pixels in logarithmic (top) and linear (bottom) scale for the cross-spectral learned feature methods on both pipelines. The whiskers extend to all data points.