MultiTaskVIF: Segmentation-oriented visible and infrared image fusion via multi-task learning

TL;DR

This work addresses the inefficiency of cascade training in segmentation-oriented VIF by introducing MultiTaskVIF, a unified framework that embeds semantic segmentation supervision directly into the fusion model via a dual-branch Multi-Task Head (MTH). By sharing a backbone and enabling hierarchical interaction between semantic and fusion features through an HIA-F module, the approach achieves superior fusion quality and segmentation performance with a single-stage, single-model training paradigm. Extensive experiments on the FMB and MSRS datasets demonstrate state-of-the-art mIoU and competitive fusion metrics, validating the practical impact of direct semantic supervision for VIF. The proposed framework generalizes across multiple backbones and AO VIF methods, offering a memory-efficient, scalable path toward application-oriented VIF.

Abstract

Visible and infrared image fusion (VIF) has attracted significant attention in recent years. Traditional VIF methods primarily focus on generating fused images with high visual quality, while recent advancements increasingly emphasize incorporating semantic information into the fusion model during training. However, most existing segmentation-oriented VIF methods adopt a cascade structure comprising separate fusion and segmentation models, leading to increased network complexity and redundancy. This raises a critical question: can we design a more concise and efficient structure to integrate semantic information directly into the fusion model during training-Inspired by multi-task learning, we propose a concise and universal training framework, MultiTaskVIF, for segmentation-oriented VIF models. In this framework, we introduce a multi-task head decoder (MTH) to simultaneously output both the fused image and the segmentation result during training. Unlike previous cascade training frameworks that necessitate joint training with a complete segmentation model, MultiTaskVIF enables the fusion model to learn semantic features by simply replacing its decoder with MTH. Extensive experimental evaluations validate the effectiveness of the proposed method. Our code will be released upon acceptance.

Figures (5)

• Figure 1: Existing cascade training framework vs. MultiTaskVIF. Both frameworks can be used to train segmentation-oriented VIF models. However, our MultiTaskVIF, which incorporates a multi-task head (MTH), is more concise as it eliminates the need for an additional full segmentation model.
• Figure 2: The workflow of proposed MultiTaskVIF training framework for AO VIF. Notably, MultiTaskVIF exhibits strong generalizability and is compatible with most existing VIF methods ma2022swinfusionzhao2024equivariantliu2023segmiftang2022image. Moreover, through this network architecture and training strategy, it effectively enhances the performance of VIF models in both image fusion and semantic segmentation tasks.
• Figure 3: Qualitative segmentation on the FMB dataset.
• Figure 4: Qualitative segmentation on the MSRS dataset.
• Figure 5: Feature maps of the two branches within MTH.