IRisPath: Enhancing Costmap for Off-Road Navigation with Robust IR-RGB Fusion for Improved Day and Night Traversability

TL;DR

IRisPath tackles robust off-road traversability under varying illumination by fusing RGB and LWIR imagery. The approach employs a dual-branch architecture with velocity-conditioned, patch-based fusion and a self-supervised cost function, trained on a new day–night dataset and supported by a targetless RGB–IR–LiDAR calibration method. Experimental results show that fusion outperforms single-modality baselines in both day and night, producing more accurate and stable traversability costmaps. These contributions advance reliable all-condition off-road navigation and provide datasets and calibration tools to spur further research.

Abstract

Autonomous off-road navigation is required for applications in agriculture, construction, search and rescue and defence. Traditional on-road autonomous methods struggle with dynamic terrains, leading to poor vehicle control in off-road conditions. Recent deep-learning models have used perception sensors along with kinesthetic feedback for navigation on such terrains. However, this approach has out-of-domain uncertainty. Factors like change in time of day and weather impacts the performance of the model. We propose a multi modal fusion network "IRisPath" capable of using Thermal and RGB images to provide robustness against dynamic weather and light conditions. To aid further works in this domain, we also open-source a day-night dataset with Thermal and RGB images along with pseudo-labels for traversability. In order to co-register for fusion model we also develop a novel method for targetless extrinsic calibration of Thermal, LiDAR and RGB cameras with translation accuracy of +/-1.7cm and rotation accuracy of +/-0.827degrees.

Figures (8)

• Figure 1: Various sensors mounted on Copernicus for the off-road dataset collection.
• Figure 2: IR-RGB Calibration Process to obtain extrinsic transformation parameters
• Figure 3: Overview of the proposed early fusion approach. The inputs to the model are in the purple boxes. To get both the input image in the same frame we apply homographic projection onto the RGB image so that we get matching image pairs of RGB and IR. Then we first split both input images into multiple smaller images on which model predicts a traversability score which are then combined and shown as a costmap.
• Figure 4: System configuration for dataset collection on the Copernicus vehicle. We initially record all the sensors and actuators as a rosbag and further process them to other formats.
• Figure 5: The RGB image is being transformed into the IR image frame using the calculated extrinsic parameters.