Towards RGB-NIR Cross-modality Image Registration and Beyond

TL;DR

The paper tackles RGB-NIR cross-modality image registration by first introducing the RGB-NIR-IRegis benchmark with the RGB-NIR-IRAP annotation pipeline to enable fair evaluation under viewpoint variations. It analyzes gradient-based inconsistencies between visible and infrared images and shows these discrepancies degrade registration performance. To address this, the authors present SGFormer, a transformer-based baseline that injects high-level semantic guidance through a Semantic Injection Module and enforces semantically-consistent descriptors via Semantic Triplet Loss, achieving state-of-the-art results on the new benchmark and on general RGB registration tasks. The work offers a practical benchmark for robust RGB-NIR registration and demonstrates the value of semantic guidance in mitigating cross-modality feature misalignment, with implications for downstream tasks like surveillance, autonomous driving, and low-light imaging.

Abstract

This paper focuses on the area of RGB(visible)-NIR(near-infrared) cross-modality image registration, which is crucial for many downstream vision tasks to fully leverage the complementary information present in visible and infrared images. In this field, researchers face two primary challenges - the absence of a correctly-annotated benchmark with viewpoint variations for evaluating RGB-NIR cross-modality registration methods and the problem of inconsistent local features caused by the appearance discrepancy between RGB-NIR cross-modality images. To address these challenges, we first present the RGB-NIR Image Registration (RGB-NIR-IRegis) benchmark, which, for the first time, enables fair and comprehensive evaluations for the task of RGB-NIR cross-modality image registration. Evaluations of previous methods highlight the significant challenges posed by our RGB-NIR-IRegis benchmark, especially on RGB-NIR image pairs with viewpoint variations. To analyze the causes of the unsatisfying performance, we then design several metrics to reveal the toxic impact of inconsistent local features between visible and infrared images on the model performance. This further motivates us to develop a baseline method named Semantic Guidance Transformer (SGFormer), which utilizes high-level semantic guidance to mitigate the negative impact of local inconsistent features. Despite the simplicity of our motivation, extensive experimental results show the effectiveness of our method.

Figures (7)

• Figure 1: Overview of our framework. As shown in the left part of the figure, with the proposed RGB-NIRImage Registration Annotation Pipeline (dubbed RGB-NIR-IRAP), we present the largest-scale RGB-NIRImage Registration (dubbed RGB-NIR-IRegis) benchmark featuring correct annotations (green lines) and viewpoint variations (unaligned RGB-NIR image pairs). This is in comparison to previous benchmarks shen2014multibrown2011multi, where factually correct matching correspondences are annotated as errors (red lines) and RGB-NIR image pairs are limited to restricted viewpoint variations. Utilizing the proposed RGB-NIR-IRegis benchmark, we fairly show that existing registration methods perform poorly, e.g, DASC kim2015dasckim2016dasc obtained AUC @3px of only 0.6 when assessed in the RGB-NIR cross-modality and cross-viewpoint scenario. This motivates us to analyze the causes for such poor performance and further develop a baseline method called SGFormer, which achieved new SOTA results for the task of RGB-NIR cross-modality image registration.
• Figure 2: The proposed RGB-NIRImage Registration Annotation Pipeline (dubbed RGB-NIR-IRAP) for collecting RGB-NIR image pairs with viewpoint variations.
• Figure 3: Examples of RGB-NIR gradient inconsistency. RGB image patches are converted to grayscale for comparison. The yellow arrow represents the gradient of the pixel with its length indicating the magnitude. Patches show the inconsistency between RGB-NIR image pairs in the gradient A) orientation; B) both magnitude and orientation.
• Figure 4: Gradient distribution comparison using Bivariate Gaussian Distribution maps. Compared with the gradient distribution shift between visible images with illumination variations in the same sensor (Fig. \ref{['Hpatch']}), there existed significant gradient distribution differences between visible and infrared images (Fig. \ref{['EPFL']} and \ref{['RGB-NIR-IMatch']}), which were more evident in our RGB-NIR-IRegis benchmark.
• Figure 5: RGB-NIR gradient inconsistency impact comparisons. The X-axis and Y-axis represent the gradient inconsistency metric $Q$ and the prediction deviation $EPE$. Results show that as the gradient inconsistency increased, the registration performance of current methods declined. Besides, the negative impact of gradient inconsistency of RGB-NIR cross-modality (RGB-NIR-IRegis) caused a more significant performance drop trend than that of RGB-RGB cross-illumination (HPatches) in general (i.e., having sharper slopes).