Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion

TL;DR

This work tackles misalignment in infrared-visible image fusion by introducing B-SR, a self-supervised bidirectional registration framework that forgoes translation-based alignment. It leverages a proxy data generator (PDG) and inverse PDG (IPDG) to produce global and local alignment signals, and enforces cross-modal edge consistency with a neighborhood dynamic alignment loss while jointly training a reconstruction module to support fusion. Key contributions include the PDG/IPDG mechanism, bi-directional deformation fields ($\phi_p$ and $\phi_n$) for intra-branch and inter-branch registration, and a comprehensive loss design ($\mathcal{L}_{nda}$, $\mathcal{L}_{epr}$, $\mathcal{L}_{ss}$, $\mathcal{L}_{smooth}$, $\mathcal{L}_{recp}$, $\mathcal{L}_{fusion}$) that yields robust alignment and high-quality fusion on misaligned data. Experimental results on RoadScene, DroneVehicle, and TNO demonstrate superior registration and fusion performance compared with state-of-the-art methods, with improved edge alignment and detail preservation under varying misalignment degrees. The approach offers a practical, translation-free solution for real-world multi-modal fusion systems, with public availability of code to foster reproducibility and further development.

Abstract

Acquiring accurately aligned multi-modal image pairs is fundamental for achieving high-quality multi-modal image fusion. To address the lack of ground truth in current multi-modal image registration and fusion methods, we propose a novel self-supervised \textbf{B}i-directional \textbf{S}elf-\textbf{R}egistration framework (\textbf{B-SR}). Specifically, B-SR utilizes a proxy data generator (PDG) and an inverse proxy data generator (IPDG) to achieve self-supervised global-local registration. Visible-infrared image pairs with spatially misaligned differences are aligned to obtain global differences through the registration module. The same image pairs are processed by PDG, such as cropping, flipping, stitching, etc., and then aligned to obtain local differences. IPDG converts the obtained local differences into pseudo-global differences, which are used to perform global-local difference consistency with the global differences. Furthermore, aiming at eliminating the effect of modal gaps on the registration module, we design a neighborhood dynamic alignment loss to achieve cross-modal image edge alignment. Extensive experiments on misaligned multi-modal images demonstrate the effectiveness of the proposed method in multi-modal image alignment and fusion against the competing methods. Our code will be publicly available.

Figures (11)

• Figure 1: (a) Self-registration paradigm in B-SR. (b) The first column shows infrared and visible image pairs with 5 pixels difference in the horizontal direction, where the yellow dashed line denotes the horizontal difference between different modalities. Compared with existing alignment-based fusion methods, B-SR effectively addresses the issue of misalignment between modalities.
• Figure 2: Architecture of the B-SR method. (a) The training process of the bi-directional self-registration framework is designed to align multi-modal images by leveraging both intra-branch bi-directional alignment constraints and inter-branch alignment results simultaneously. The inter-branch alignment is achieved through PDG and IPDG, while the intra-branch alignment is achieved through the bi-directional deformation field. The reconstruction module through joint registration-reconstruction optimization, will fix the feature extraction parameters, thereby providing an efficient feature extraction layer for subsequent fusion. (b) The training process of the fusion module.
• Figure 3: Comparisons of fusion results with a 5-pixel difference in the horizontal direction for "$FLIR\_06993$" in the RoadScene dataset.
• Figure 4: Comparison of results in a DroneVehicle dataset based on drone views.
• Figure 5: The analysis of the ablation experiment was conducted using the RoadScene dataset.