LLHA-Net: A Hierarchical Attention Network for Two-View Correspondence Learning

TL;DR

LLHA-Net tackles the challenge of two-view correspondence learning under high outlier ratios by introducing a hierarchical, multi-stage architecture. It combines Layer-by-Layer Channel Fusion (LLF), Hierarchical Attention (HA), and Permutation Invariance Hierarchical Attention (PIHA) to progressively preserve and fuse semantic information across stages, enabling robust inlier/outlier discrimination and accurate essential-matrix estimation via a weighted eight-point method. The network is trained with a hybrid loss that jointly optimizes inlier classification and geometric consistency, and is evaluated on YFCC100M and SUN3D, where it achieves state-of-the-art precision and F-scores for outlier removal and strong camera pose estimation performance, including in unknown scenes. This work advances practical 3D reconstruction and image registration applications by enabling more reliable feature correspondences in challenging real-world conditions, while acknowledging higher computational cost and potential misclassification of some inliers as outliers as areas for future optimization.

Abstract

Establishing the correct correspondence of feature points is a fundamental task in computer vision. However, the presence of numerous outliers among the feature points can significantly affect the matching results, reducing the accuracy and robustness of the process. Furthermore, a challenge arises when dealing with a large proportion of outliers: how to ensure the extraction of high-quality information while reducing errors caused by negative samples. To address these issues, in this paper, we propose a novel method called Layer-by-Layer Hierarchical Attention Network, which enhances the precision of feature point matching in computer vision by addressing the issue of outliers. Our method incorporates stage fusion, hierarchical extraction, and an attention mechanism to improve the network's representation capability by emphasizing the rich semantic information of feature points. Specifically, we introduce a layer-by-layer channel fusion module, which preserves the feature semantic information from each stage and achieves overall fusion, thereby enhancing the representation capability of the feature points. Additionally, we design a hierarchical attention module that adaptively captures and fuses global perception and structural semantic information using an attention mechanism. Finally, we propose two architectures to extract and integrate features, thereby improving the adaptability of our network. We conduct experiments on two public datasets, namely YFCC100M and SUN3D, and the results demonstrate that our proposed method outperforms several state-of-the-art techniques in both outlier removal and camera pose estimation. Source code is available at http://www.linshuyuan.com.

Figures (9)

• Figure 1: LLHA-Net: Overall network architecture and feature integration modules. The upper part illustrates the feature extraction architecture, while the lower part shows the overall network. The feature integration architecture includes multiple PIHA modules.
• Figure 2: LLF module: layer-by-layer channel fusion module.
• Figure 3: HA module: Hierarchical attention module uses the attention mechanism to adaptively fuse the information extracted from different hierarchical structures, while the LLF module fuses the information of each channel.
• Figure 4: The weight variations of global perception information and local semantic information in the HA module are shown over 500,000 iterations. The horizontal axis represents the number of iterations, measured in units of ten thousand.
• Figure 5: Some visualization results obtained using 5 competing methods on the YFCC100M dataset. Each row corresponds to a different method: the $1^{st}$ to $5^{th}$ rows represent OA-Net oanet, T-Net T-Net, PESA zhong2022pesa, MSA-Net MSA-net, and the proposed LLHA-Net, respectively. Red indicates remaining incorrect correspondences, while green indicates correct correspondences.