All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark

TL;DR

Experimental results show that the proposed all-weather MMIF model excels in detail recovery and multi-modality feature extraction, and a large-scale multi-modality dataset is constructed, covering various degradation levels and diverse scenes to thoroughly evaluate image fusion methods in adverse weather.

Abstract

Multi-modality image fusion (MMIF) combines complementary information from different image modalities to provide a more comprehensive and objective interpretation of scenes. However, existing MMIF methods lack the ability to resist different weather interferences in real-world scenes, preventing them from being useful in practical applications such as autonomous driving. To bridge this research gap, we proposed an all-weather MMIF model. Achieving effective multi-tasking in this context is particularly challenging due to the complex and diverse nature of weather conditions. A key obstacle lies in the 'black box' nature of current deep learning architectures, which restricts their multi-tasking capabilities. To overcome this, we decompose the network into two modules: a fusion module and a restoration module. For the fusion module, we introduce a learnable low-rank representation model to decompose images into low-rank and sparse components. This interpretable feature separation allows us to better observe and understand images. For the restoration module, we propose a physically-aware clear feature prediction module based on an atmospheric scattering model that can deduce variations in light transmittance from both scene illumination and reflectance. We also construct a large-scale multi-modality dataset with 100,000 image pairs across rain, haze, and snow conditions, covering various degradation levels and diverse scenes to thoroughly evaluate image fusion methods in adverse weather. Experimental results in both real-world and synthetic scenes show that the proposed algorithm excels in detail recovery and multi-modality feature extraction. The code is available at https://github.com/ixilai/AWFusion.

Figures (11)

• Figure 1: The architecture of the proposed all-weather MMIF framework.
• Figure 2: Network structure of RFB, IFB, and SFB. $q_k \in \{d_k, m_k, s_{I,k}, s_{V,k}\}$ and $Q_j \in \{D_j, M_j, S_j^I, S_j^V\}$.
• Figure 3: Schematic representation of real-world and synthetic scene data. In this case, the data for the real scene is from AWMM-100k and the data for the synthesized scene is from the RoadScene dataset.
• Figure 4: Partial real multi-modality data from AWMM-100k.
• Figure 5: Comparison of fusion results obtained by the proposed algorithm under rain weather, and the results of the comparison methods under ideal condition.