Robust Fusion Controller: Degradation-aware Image Fusion with Fine-grained Language Instructions

TL;DR

This paper introduces a Robust Fusion Controller (RFC) that enables degradation-aware image fusion guided by fine-grained language instructions. RFC converts language into a functional condition and a spatial condition, fuses them into a composite control prior via a multi-condition coupling network, and embeds this prior into a hybrid attention fusion network with feature-wise affine modulation. It employs a degradation-aware reconstruction loss and a language-feature alignment loss to align outputs with language prompts, achieving robust performance under spatially varying composite degradations, including flare. Experimental results on multiple public datasets demonstrate superior perceptual quality and improved downstream semantic tasks, with strong generalization to new fusion scenarios. The approach offers a flexible, interpretable framework for controlling multi-modal fusion in harsh, real-world environments.

Abstract

Current image fusion methods struggle to adapt to real-world environments encompassing diverse degradations with spatially varying characteristics. To address this challenge, we propose a robust fusion controller (RFC) capable of achieving degradation-aware image fusion through fine-grained language instructions, ensuring its reliable application in adverse environments. Specifically, RFC first parses language instructions to innovatively derive the functional condition and the spatial condition, where the former specifies the degradation type to remove, while the latter defines its spatial coverage. Then, a composite control priori is generated through a multi-condition coupling network, achieving a seamless transition from abstract language instructions to latent control variables. Subsequently, we design a hybrid attention-based fusion network to aggregate multi-modal information, in which the obtained composite control priori is deeply embedded to linearly modulate the intermediate fused features. To ensure the alignment between language instructions and control outcomes, we introduce a novel language-feature alignment loss, which constrains the consistency between feature-level gains and the composite control priori. Extensive experiments on publicly available datasets demonstrate that our RFC is robust against various composite degradations, particularly in highly challenging flare scenarios.

Figures (13)

• Figure 1: Comparisons between our RFC and existing fusion methods in the degraded scenario.
• Figure 2: The overall framework of our proposed RFC.
• Figure 3: The architecture of multi-condition coupling module.
• Figure 4: The architecture of hybrid attention fusion module.
• Figure 5: Schematic diagram of the alignment mechanism between feature-level gains and language instructions.