FusionFM: All-in-One Multi-Modal Image Fusion with Flow Matching

TL;DR

FusionFM addresses the scalability and efficiency challenges of multi-modal image fusion by reframing MMIF as direct probabilistic transport via flow matching. It introduces a two-stage pseudo-ground-truth generation using multiple state-of-the-art priors and a dedicated Fusion Refiner to produce high-quality supervisory signals. A continual-learning backbone combining elastic weight consolidation and experience replay preserves cross-task knowledge while enabling continual fusion across modalities. The approach delivers competitive results across IVF, MIF, MEF, and MFF tasks, with significantly improved sampling efficiency and a lightweight model suitable for real-time deployment, and it demonstrates strong downstream benefits in semantic segmentation and object detection.

Abstract

Current multi-modal image fusion methods typically rely on task-specific models, leading to high training costs and limited scalability. While generative methods provide a unified modeling perspective, they often suffer from slow inference due to the complex sampling trajectories from noise to image. To address this, we formulate image fusion as a direct probabilistic transport from source modalities to the fused image distribution, leveraging the flow matching paradigm to improve sampling efficiency and structural consistency. To mitigate the lack of high-quality fused images for supervision, we collect fusion results from multiple state-of-the-art models as priors, and employ a task-aware selection function to select the most reliable pseudo-labels for each task. We further introduce a Fusion Refiner module that employs a divide-and-conquer strategy to systematically identify, decompose, and enhance degraded components in selected pseudo-labels. For multi-task scenarios, we integrate elastic weight consolidation and experience replay mechanisms to preserve cross-task performance and enhance continual learning ability from both parameter stability and memory retention perspectives. Our approach achieves competitive performance across diverse fusion tasks, while significantly improving sampling efficiency and maintaining a lightweight model design. The code will be available at: https://github.com/Ist-Zhy/FusionFM.

Figures (7)

• Figure 1: Motivation of our work. (a) Existing methods rely on “one-task-one-model” and suffer from limited generalization in joint training. (b) Compares traditional diffusion sampling with the efficient and adaptable straight sampling paths of flow matching. (c) Proposed FusionFM Framework.
• Figure 2: Overview of our training pipeline. The latent input $x_0$ is formed by adding two modality-specific source images $x_0^A$ and $x_0^B$. A set of fusion candidates is generated by multiple pretrained fusion models $\mathcal{M}$ and scored by the task-aware selector $\varphi$. The top-ranked result is further refined by the Fusion Refiner network to produce the pseudo ground truth $x_1$. Flow Matching is trained to learn the vector field from $x_0$ to $x_1$, conditioned on $(x_0^A, x_0^B)$.
• Figure 3: Qualitative results of previous methods and proposed method. From top to bottom, they are IVF, MIF, MEF, and MFF.
• Figure 4: Visualization results of semantic segmentation in two scenes on the MSRS dataset.
• Figure 5: Visualization results of semantic segmentation in two scenes on the FMB dataset.