FuseFormer: A Transformer for Visual and Thermal Image Fusion
Aytekin Erdogan, Erdem Akagündüz
TL;DR
The paper tackles RGB-IR image fusion without reliable ground truth, addressing the bias introduced by SSIM-based losses. It introduces FuseFormer, a dual-branch fusion block that combines CNN-based local feature extraction with Vision Transformer-based global context, integrated into a two-stage training pipeline. A novel fusion loss $L_{fuse}=L_{feature}+\alpha L_{\overline{ssim}}$, with $L_{\overline{ssim}}=[1-SSIM(I_f,I_v)]^2+[1-SSIM(I_f,I_i)]^2$ and $L_{feature}=\sum_{m=1}^{M} \omega^m \| \phi_f^m-(\omega_{vi}\phi_{vi}^m+\omega_{ir}\phi_{ir}^m)\|_F^2$, balances cross-modal fidelity and global-context integration. Across MS-COCO, RoadScene, and TNO benchmarks, FuseFormer achieves competitive or state-of-the-art performance, with notable qualitative improvements in low-light scenarios, suggesting broader applicability to universal multi-spectral fusion.
Abstract
Due to the lack of a definitive ground truth for the image fusion problem, the loss functions are structured based on evaluation metrics, such as the structural similarity index measure (SSIM). However, in doing so, a bias is introduced toward the SSIM and, consequently, the input visual band image. The objective of this study is to propose a novel methodology for the image fusion problem that mitigates the limitations associated with using classical evaluation metrics as loss functions. Our approach integrates a transformer-based multi-scale fusion strategy that adeptly addresses local and global context information. This integration not only refines the individual components of the image fusion process but also significantly enhances the overall efficacy of the method. Our proposed method follows a two-stage training approach, where an auto-encoder is initially trained to extract deep features at multiple scales in the first stage. For the second stage, we integrate our fusion block and change the loss function as mentioned. The multi-scale features are fused using a combination of Convolutional Neural Networks (CNNs) and Transformers. The CNNs are utilized to capture local features, while the Transformer handles the integration of general context features. Through extensive experiments on various benchmark datasets, our proposed method, along with the novel loss function definition, demonstrates superior performance compared to other competitive fusion algorithms.