Focus What Matters: Matchability-Based Reweighting for Local Feature Matching

TL;DR

This work introduces a matchability-aware reweighting framework for Transformer-based local feature matching. It explicitly predicts per-pixel matchability maps and uses them to bias attention logits before softmax and to rescale value features after attention, enabling the model to focus on reliable regions. The method combines a CNN backbone, coarse-to-fine matching, and a multi-term loss that includes binary matchability supervision, achieving state-of-the-art results on MegaDepth, ScanNet, and HPatches for both relative pose estimation and image matching. The proposed approach enhances robustness to appearance changes and clutter, with practical implications for SfM, SLAM, and visual localization tasks.

Abstract

Since the rise of Transformers, many semi-dense matching methods have adopted attention mechanisms to extract feature descriptors. However, the attention weights, which capture dependencies between pixels or keypoints, are often learned from scratch. This approach can introduce redundancy and noisy interactions from irrelevant regions, as it treats all pixels or keypoints equally. Drawing inspiration from keypoint selection processes, we propose to first classify all pixels into two categories: matchable and non-matchable. Matchable pixels are expected to receive higher attention weights, while non-matchable ones are down-weighted. In this work, we propose a novel attention reweighting mechanism that simultaneously incorporates a learnable bias term into the attention logits and applies a matchability-informed rescaling to the input value features. The bias term, injected prior to the softmax operation, selectively adjusts attention scores based on the confidence of query-key interactions. Concurrently, the feature rescaling acts post-attention by modulating the influence of each value vector in the final output. This dual design allows the attention mechanism to dynamically adjust both its internal weighting scheme and the magnitude of its output representations. Extensive experiments conducted on three benchmark datasets validate the effectiveness of our method, consistently outperforming existing state-of-the-art approaches.

Figures (5)

• Figure 1: Comparison between EfficientLoFTR (above) and the proposal (below). Green color indicates epipolar error below $5 \times 10^{-4}$.
• Figure 2: The illustration of conceptual difference between two feature matching pipelines. (a) Existing matching pipeline. (b) The matching pipeline of the proposal. Our method adaptively modify the attention weights, making it more focus on the foreground, which is often occupied with matchable points.
• Figure 3: Visualization of cross attention heatmaps obtained by different methods. The red color means the high concentration. Notice that the proposal precisely concentrates on the specific corresponding region, whereas ELoFTR has a tendency to diffuse among wider spatial areas.
• Figure 4: The overview of the proposed matching pipeline. Matchability maps are estimated through the correlation of features from the CNN backbone. These maps are then fed into the attention module to reweight attention weights. $R(\cdot)$ denotes the RoPE positional encoding. For clarity, we present the reweighting attention module in the context of cross-attention. The formulation for self-attention is analogous, differing only in that the input features are identical.
• Figure 5: Graphical illustration on how attention weights are modified based on the estimated saliency map. The color at the right denotes the matchability score of each pixel, from top (high) to down (low).