BSAFusion: A Bidirectional Stepwise Feature Alignment Network for Unaligned Medical Image Fusion

TL;DR

The work tackles unaligned multimodal medical image fusion by introducing a single stage framework that integrates registration and fusion. It combines Modality Discrepancy-Free Feature Representation MDF-FR, Bidirectional Stepwise Feature Alignment BSFA and Multi-Modal Feature Fusion MMFF within a shared encoder to align and fuse images without separate registration. Key innovations include modality heads for global representation, a bidirectional deformation field progression with path independence, and FusionBLK driven fusion, all guided by dedicated losses. Across CT, MRI, PET and SPECT datasets, the method delivers superior fusion quality and robust alignment with competitive model efficiency, highlighting potential for clinically efficient image fusion systems. The authors also provide public code to reproduce and extend the approach.

Abstract

If unaligned multimodal medical images can be simultaneously aligned and fused using a single-stage approach within a unified processing framework, it will not only achieve mutual promotion of dual tasks but also help reduce the complexity of the model. However, the design of this model faces the challenge of incompatible requirements for feature fusion and alignment; specifically, feature alignment requires consistency among corresponding features, whereas feature fusion requires the features to be complementary to each other. To address this challenge, this paper proposes an unaligned medical image fusion method called Bidirectional Stepwise Feature Alignment and Fusion (BSFA-F) strategy. To reduce the negative impact of modality differences on cross-modal feature matching, we incorporate the Modal Discrepancy-Free Feature Representation (MDF-FR) method into BSFA-F. MDF-FR utilizes a Modality Feature Representation Head (MFRH) to integrate the global information of the input image. By injecting the information contained in MFRH of the current image into other modality images, it effectively reduces the impact of modality differences on feature alignment while preserving the complementary information carried by different images. In terms of feature alignment, BSFA-F employs a bidirectional stepwise alignment deformation field prediction strategy based on the path independence of vector displacement between two points. This strategy solves the problem of large spans and inaccurate deformation field prediction in single-step alignment. Finally, Multi-Modal Feature Fusion block achieves the fusion of aligned features. The experimental results across multiple datasets demonstrate the effectiveness of our method. The source code is available at https://github.com/slrl123/BSAFusion.

Figures (10)

• Figure 1: Paradigm of existing unaligned image fusion methods compared to that of our method.
• Figure 2: Overall framework of the proposed method. The unaligned multimodal medical image pairs $\{\bm{I}_{A}, \bm{I}_{B}\}$ are processed through the MDF-FR module, yielding features $\{\bm{F}_{A}^{s}, \bm{F}_{B}^{s}\}$ and $\{\hat{\bm{F}}_{A}, \hat{\bm{F}}_{B}\}$. Additionally, modality-specific feature representation heads, denoted as $\bm{\hat{f}}_{A}$ and $\bm{\hat{f}}_{B}$, are generated. These heads are utilized to minimize the modality disparities between $\{\bm{\hat{F}}_{A}, \bm{\hat{F}}_{B}\}$. Within the BSFA, a progressive deformation field prediction, denoted as $\bm{\phi}_{\overrightarrow{AB}}$, is carried out based on the modality-discrepancy-mitigated features $\{\bm{\bar{F}}_{A}, \bm{\bar{F}}_{B}\}$. Finally, the features $\{\bm{\bar{F}}_{A}, \bm{\bar{F}}_{B}\}$, $\{\bm{F}_{A}^{s}, \bm{F}_{B}^{s}\}$, along with the predicted deformation field $\bm{\phi}_{\overrightarrow{AB}}$, are fed into the MMFF module to generate the final fused result.
• Figure 3: Schematic of different alignment methods.
• Figure 4: Visual Comparison of Fusion Results: Joint Registration and Fusion Method vs. Our Method. The first column shows the deformed image to be fused, the second column displays the corresponding label, and the third column presents the MRI image to be fused. Columns 4 to 9 show the results obtained by different fusion methods.
• Figure 5: Comparison of objective evaluation results: Joint registration and fusion vs. the proposed method. The black line denotes the median, and the red line denotes the mean.