ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Prompts

TL;DR

ControlFusion addresses robust infrared-visible image fusion under composite degradations by combining a physics-informed degradation model with a prompt-driven restoration-fusion pipeline. It introduces Stage I text-visual prompt alignment via CLIP and a spatial-frequency visual adapter, and Stage II a hierarchical Transformer with a Prompt-Modulated Module to adapt fusion features to degradation prompts. Key contributions include a physics-driven degradation model $D_m = \mathcal{P}_s(\mathcal{P}_w(\mathcal{P}_i(I_m)))$, a SFVA for automatic degradation cue extraction, and a PMM-driven, cross-modal fusion network that yields $I_f = \mathcal{N}_{rf}(I_{ir}, I_{vi}, p|\Omega)$ with losses $\mathcal{L}_{II}$ balancing intensity, structure, edges, and color. Experiments on the synthetic DDL-12 dataset and real-world benchmarks demonstrate state-of-the-art degradation handling and competitive fusion quality, with flexible degree control via degradation prompts and efficient inference.

Abstract

Current image fusion methods struggle to address the composite degradations encountered in real-world imaging scenarios and lack the flexibility to accommodate user-specific requirements. In response to these challenges, we propose a controllable image fusion framework with language-vision prompts, termed ControlFusion, which adaptively neutralizes composite degradations. On the one hand, we develop a degraded imaging model that integrates physical imaging mechanisms, including the Retinex theory and atmospheric scattering principle, to simulate composite degradations, thereby providing potential for addressing real-world complex degradations from the data level. On the other hand, we devise a prompt-modulated restoration and fusion network that dynamically enhances features with degradation prompts, enabling our method to accommodate composite degradation of varying levels. Specifically, considering individual variations in quality perception of users, we incorporate a text encoder to embed user-specified degradation types and severity levels as degradation prompts. We also design a spatial-frequency collaborative visual adapter that autonomously perceives degradations in source images, thus eliminating the complete dependence on user instructions. Extensive experiments demonstrate that ControlFusion outperforms SOTA fusion methods in fusion quality and degradation handling, particularly in countering real-world and compound degradations with various levels. The source code is publicly available at https://github.com/Linfeng-Tang/ControlFusion.

Figures (11)

• Figure 1: Comparison across real-world, composite degradation, and varying degradation levels.
• Figure 2: The overall framework of our controllable image fusion network.
• Figure 3: The visualization of various types of degradation in the spatial and frequency domains.
• Figure 4: Visualization of fusion results under different degradation scenarios.
• Figure 5: Generalization results under real-world degradation scenarios.