A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware

TL;DR

This work addresses robust road-terrain perception for autonomous vehicles under varying lighting by introducing IMF, a multi-modal fusion network that explicitly leverages illumination features to reweight exteroceptive and proprioceptive inputs. The architecture combines an illumination perception sub-network with modality-specific extractors and an SE-guided fusion module, trained with a joint loss ${L= L_r + \lambda L_i}$ and a pre-training stage for illumination. The authors build two multi-modal datasets (camera+accelerometer and camera+intelligent-tire) and demonstrate that IMF outperforms state-of-the-art baselines and single-modality methods, especially under night-time conditions, while remaining computationally efficient. The findings highlight the practical potential of illumination-aware fusion for robust AV perception in real-world driving and point to future work on extreme weather and additional road-surface characteristics.

Abstract

Road terrains play a crucial role in ensuring the driving safety of autonomous vehicles (AVs). However, existing sensors of AVs, including cameras and Lidars, are susceptible to variations in lighting and weather conditions, making it challenging to achieve real-time perception of road conditions. In this paper, we propose an illumination-aware multi-modal fusion network (IMF), which leverages both exteroceptive and proprioceptive perception and optimizes the fusion process based on illumination features. We introduce an illumination-perception sub-network to accurately estimate illumination features. Moreover, we design a multi-modal fusion network which is able to dynamically adjust weights of different modalities according to illumination features. We enhance the optimization process by pre-training of the illumination-perception sub-network and incorporating illumination loss as one of the training constraints. Extensive experiments demonstrate that the IMF shows a superior performance compared to state-of-the-art methods. The comparison results with single modality perception methods highlight the comprehensive advantages of multi-modal fusion in accurately perceiving road terrains under varying lighting conditions. Our dataset is available at: https://github.com/lindawang2016/IMF.

Figures (14)

• Figure 1: The proposed illumintion-aware multi-modal fusion network
• Figure 2: The illumination perception module
• Figure 3: The fusion module with illumination features
• Figure 4: The experiment vehicle and the sensors installation.
• Figure 5: The raw acceleration data under different illumination conditions:(a),(d),(g): asphalt at noon, dusk and night; (b),(e),(h): gravel at noon, dusk and night; (c),(f),(i): cement at noon, dusk and night.