Image Fusion via Vision-Language Model

TL;DR

This paper tackles the limitation of image fusion methods that rely predominantly on visual cues by introducing FILM, a fusion framework that harnesses textual information extracted from source images through a vision-language model to guide fusion. The approach generates semantic prompts from images, feeds them to ChatGPT to obtain descriptive text, fuses these descriptions in the textual domain, and uses cross-attention to steer visual feature fusion before decoding the final image. Key contributions include (i) a three-component FILM pipeline (text feature fusion, language-guided vision feature fusion, vision feature decoding), (ii) a Vision-Language Fusion (VLF) dataset with ChatGPT-generated descriptions for eight fusion datasets, and (iii) extensive experiments across IVF, MIF, MEF, and MFF showing competitive or superior results to state-of-the-art methods. The work demonstrates the potential of integrating vision-language models into image fusion, enabling deeper semantic guidance and better cross-modal information synthesis, with code and data publicly available for further research.

Abstract

Image fusion integrates essential information from multiple images into a single composite, enhancing structures, textures, and refining imperfections. Existing methods predominantly focus on pixel-level and semantic visual features for recognition, but often overlook the deeper text-level semantic information beyond vision. Therefore, we introduce a novel fusion paradigm named image Fusion via vIsion-Language Model (FILM), for the first time, utilizing explicit textual information from source images to guide the fusion process. Specifically, FILM generates semantic prompts from images and inputs them into ChatGPT for comprehensive textual descriptions. These descriptions are fused within the textual domain and guide the visual information fusion, enhancing feature extraction and contextual understanding, directed by textual semantic information via cross-attention. FILM has shown promising results in four image fusion tasks: infrared-visible, medical, multi-exposure, and multi-focus image fusion. We also propose a vision-language dataset containing ChatGPT-generated paragraph descriptions for the eight image fusion datasets across four fusion tasks, facilitating future research in vision-language model-based image fusion. Code and dataset are available at https://github.com/Zhaozixiang1228/IF-FILM.

Figures (13)

• Figure 1: Workflow for our FILM. Input images are first processed to create prompts for ChatGPT chatgpt, which then generate detailed textual descriptions. These descriptions help to get fused textual features via the frozen BLIP2 DBLP:conf/icml/0008LSH23 model. Then, these textual features are fused and guide the extraction and fusion of visual features via cross-attention, enhancing contextual understanding with text-based semantic information. Finally, the fusion image is output by the image decoder.
• Figure 2: Workflow for our FILM, which encompasses three components: text paragraph generation and text feature fusion, language-guided vision feature fusion via cross attention and vision feature decoding, corresponding to the first, second, and third columns.
• Figure 3: Visualization of the VLF dataset creation process and representative data displays.
• Figure 4: Visualization comparison of the fusion results in the infrared-visible image fusion task.
• Figure 5: Visualization comparison of the fusion results in the medical image fusion task.