CtrlFuse: Mask-Prompt Guided Controllable Infrared and Visible Image Fusion

TL;DR

CtrlFuse addresses the limitation of static, task-fixed multimodal fusion by enabling interactive, mask-prompt guided fusion of infrared and visible images. It introduces a Reference Prompt Encoder to extract task-specific semantic prompts and a Prompt-Semantic Fusion Module to inject these semantics into fusion features, guided by a frozen Segment Anything Model. The framework jointly optimizes fusion and segmentation losses, achieving mutual enhancement between fusion quality and downstream perception accuracy across multiple datasets, with strong controllability demonstrated through prompt-based modulation. This approach enables semantically aware, all-weather fusion suitable for intelligent unmanned systems and downstream tasks like segmentation and object detection.

Abstract

Infrared and visible image fusion generates all-weather perception-capable images by combining complementary modalities, enhancing environmental awareness for intelligent unmanned systems. Existing methods either focus on pixel-level fusion while overlooking downstream task adaptability or implicitly learn rigid semantics through cascaded detection/segmentation models, unable to interactively address diverse semantic target perception needs. We propose CtrlFuse, a controllable image fusion framework that enables interactive dynamic fusion guided by mask prompts. The model integrates a multi-modal feature extractor, a reference prompt encoder (RPE), and a prompt-semantic fusion module (PSFM). The RPE dynamically encodes task-specific semantic prompts by fine-tuning pre-trained segmentation models with input mask guidance, while the PSFM explicitly injects these semantics into fusion features. Through synergistic optimization of parallel segmentation and fusion branches, our method achieves mutual enhancement between task performance and fusion quality. Experiments demonstrate state-of-the-art results in both fusion controllability and segmentation accuracy, with the adapted task branch even outperforming the original segmentation model.

Figures (22)

• Figure 1: The importance of mask prompt-based interactive controllable image fusion for the performance of downstream application models (e.g., semantic segmentation).
• Figure 2: The architecture of CtrlFuse. CtrlFuse consists of two reference prompt encoders, two Prompt-Semantic fusion modules guided by the prompt mask, a set of infrared and visible feature encoders, and the auxiliary network.
• Figure 3: Qualitative comparisons of various methods on representative images selected from the FMB dataset.
• Figure 4: Qualitative comparisons of various methods on representative images selected from DroneVehicle dataset.
• Figure 5: Segmentation results for infrared, visible, and fused images from the MSRS dataset. The segmentation models are retrained.