Entity-Guided Multi-Task Learning for Infrared and Visible Image Fusion

TL;DR

This work tackles semantic noise and underutilization of textual information in infrared-visible image fusion by introducing Entity-Guided Multi-Task Learning (EGMT). It combines entity-level semantics extracted from LVLM captions with a parallel fusion-plus-classification framework, enabling a multi-task objective that uses entity pseudo-labels to strengthen semantic supervision. A novel entity-guided cross-modal interaction module (ECIM) facilitates fine-grained inter-visual and visual-entity feature interactions, suppressing LVLM hallucinations and improving fusion quality, as demonstrated across TNO, RoadScene, M$^{3}$FD, and MSRS with strong downstream benefits in object detection. The approach achieves superior fusion metrics and competitive multi-label classification while maintaining practical efficiency, establishing a new semantic-guided paradigm for infrared-visible image fusion.

Abstract

Existing text-driven infrared and visible image fusion approaches often rely on textual information at the sentence level, which can lead to semantic noise from redundant text and fail to fully exploit the deeper semantic value of textual information. To address these issues, we propose a novel fusion approach named Entity-Guided Multi-Task learning for infrared and visible image fusion (EGMT). Our approach includes three key innovative components: (i) A principled method is proposed to extract entity-level textual information from image captions generated by large vision-language models, eliminating semantic noise from raw text while preserving critical semantic information; (ii) A parallel multi-task learning architecture is constructed, which integrates image fusion with a multi-label classification task. By using entities as pseudo-labels, the multi-label classification task provides semantic supervision, enabling the model to achieve a deeper understanding of image content and significantly improving the quality and semantic density of the fused image; (iii) An entity-guided cross-modal interactive module is also developed to facilitate the fine-grained interaction between visual and entity-level textual features, which enhances feature representation by capturing cross-modal dependencies at both inter-visual and visual-entity levels. To promote the wide application of the entity-guided image fusion framework, we release the entity-annotated version of four public datasets (i.e., TNO, RoadScene, M3FD, and MSRS). Extensive experiments demonstrate that EGMT achieves superior performance in preserving salient targets, texture details, and semantic consistency, compared to the state-of-the-art methods. The code and dataset will be publicly available at https://github.com/wyshao-01/EGMT.

Figures (7)

• Figure 1: Illustration of (a) General image fusion approach, (b) Cascaded downstream-task image fusion approach, and (c) Textual interaction-based image fusion approach. (d) Our EGMT framework jointly leverages entity semantics and visual features to simultaneously optimize image fusion (main task) and multi-label classification (auxiliary task) through multi-task learning.
• Figure 2: Workflow of our EGMT framework. Source images are processed by BLIP-2 to generate captions, from which Flan-T5 extracts entities that are encoded into features via CLIP. Concurrently, convolutional blocks extract shallow visual features from the source images. Both features are integrated through our entity-guided cross-modal interaction module (ECIM) to produce shared representations, which are finally decoded by task-specific heads into the fused image and classification results.
• Figure 3: Architecture of the Entity-guided Cross-modal Interaction Module (ECIM). The module takes shallow visual features and entity features as inputs. The inter-visual interaction path (left) processes features through multi-head cross-attention (MCA) and multi-head self-attention (MSA) to capture intra- and inter-modal dependencies. The visual-entity interaction path (right) employs cross-modal guided hybrid attention (CGHA) to enable alternating guidance between visual and entity features.
• Figure 4: Qualitative assessment of our EGMT and the other SOTA methods across TNO, RoadScene, M$^{3}$FD, and MSRS datasets. Specifically, four representative examples, namely Kaptein_1123, FILR_06307, 03878, and 00427D are selected for visual comparison. Entities from the infrared image captions are rendered in red, and those from the visible image captions are in blue.
• Figure 5: Entity activation heat maps of our EGMT on the MSRS dataset.