MdaIF: Robust One-Stop Multi-Degradation-Aware Image Fusion with Language-Driven Semantics

TL;DR

This work tackles infrared-visible image fusion under adverse weather by addressing the degraded visibility of the color channel and the inadequacy of fixed architectures. It introduces MdaIF, a one-stop degradation-aware fusion framework that uses a Vision-Language Model-derived semantic prior to guide a degradation-aware mixture-of-experts (DMoE) and a degradation-aware channel attention module (DCAM). The semantic prior, extracted by BLIP-2 and refined into weather and scene components, informs both the adaptive routing of experts and channel modulation for robust fusion across haze, rain, and snow. Empirical results on MSRS and FMB demonstrate superior performance over state-of-the-art methods, validating the approach’s generalization and practical impact in degraded multi-modal fusion scenarios.

Abstract

Infrared and visible image fusion aims to integrate complementary multi-modal information into a single fused result. However, existing methods 1) fail to account for the degradation visible images under adverse weather conditions, thereby compromising fusion performance; and 2) rely on fixed network architectures, limiting their adaptability to diverse degradation scenarios. To address these issues, we propose a one-stop degradation-aware image fusion framework for multi-degradation scenarios driven by a large language model (MdaIF). Given the distinct scattering characteristics of different degradation scenarios (e.g., haze, rain, and snow) in atmospheric transmission, a mixture-of-experts (MoE) system is introduced to tackle image fusion across multiple degradation scenarios. To adaptively extract diverse weather-aware degradation knowledge and scene feature representations, collectively referred to as the semantic prior, we employ a pre-trained vision-language model (VLM) in our framework. Guided by the semantic prior, we propose degradation-aware channel attention module (DCAM), which employ degradation prototype decomposition to facilitate multi-modal feature interaction in channel domain. In addition, to achieve effective expert routing, the semantic prior and channel-domain modulated features are utilized to guide the MoE, enabling robust image fusion in complex degradation scenarios. Extensive experiments validate the effectiveness of our MdaIF, demonstrating superior performance over SOTA methods.

Figures (7)

• Figure 1: Our fusion results under haze, rain, and snow conditions, producing clean outputs from degraded inputs.
• Figure 2: Comparisons with previous pipelines.
• Figure 3: Overview of the proposed network architecture.
• Figure 4: The process of semantic prior extraction and its deep interaction with image features.
• Figure 5: (a) Decomposition of semantic prior into degradation prototypes across different weather types. (b) Radar chart of top 10 activated (outward) and suppressed (inward) channels per degradation prototype.