XRefine: Attention-Guided Keypoint Match Refinement

TL;DR

Sparse keypoint matches often suffer from sub-pixel misalignment that degrades downstream geometry. XRefine presents a detector-agnostic, cross-attention-based refinement that operates only on local image patches to predict sub-pixel keypoint displacements, and it scales to multi-view tracks for SfM. Across MegaDepth, ScanNet, and KITTI, XRefine consistently improves relative pose estimation and 3D triangulation with competitive runtime, outperforming prior refinement methods while generalizing across detectors. The approach is trained independently of detector-specific outputs and is open-sourced, enabling broad adoption in resource-constrained 3D vision pipelines.

Abstract

Sparse keypoint matching is crucial for 3D vision tasks, yet current keypoint detectors often produce spatially inaccurate matches. Existing refinement methods mitigate this issue through alignment of matched keypoint locations, but they are typically detector-specific, requiring retraining for each keypoint detector. We introduce XRefine, a novel, detector-agnostic approach for sub-pixel keypoint refinement that operates solely on image patches centered at matched keypoints. Our cross-attention-based architecture learns to predict refined keypoint coordinates without relying on internal detector representations, enabling generalization across detectors. Furthermore, XRefine can be extended to handle multi-view feature tracks. Experiments on MegaDepth, KITTI, and ScanNet demonstrate that the approach consistently improves geometric estimation accuracy, achieving superior performance compared to existing refinement methods while maintaining runtime efficiency. Our code and trained models can be found at https://github.com/boschresearch/xrefine.

Figures (5)

• Figure 1: Attention-guided match refinement efficiently improves relative pose estimation. Left: Exemplary matched SuperPoint superpoint keypoints on MegaDepth megadepth. The input to our model are the $11\times11$ patches within the red dotted lines. The refined keypoints of our model are presented as yellow dots. Right: Runtime and pose estimation improvement on MegaDepth (measured as relative increase in AUC5) of match refinement approaches averaged over five feature extractors: DeDoDe dedode, SIFT sift, SuperPoint superpoint, and XFeat xfeat. We compare our generalizing model to Keypt2Subpx keypt2subpx and the match refinement solution of PixSfM pixsfm. PixSfM extracts dense S2DNet S2DNet embeddings for feature-metric refinement. Depending on the use case this might be done exclusively for the refinement. Accordingly, we show the runtime of PixSfM with and without S2DNet inference.
• Figure 2: Example match refinements from our model on MegaDepth megadepth for SuperPoint superpoint keypoints. The original keypoints are shown as red dots. In the magnified patches, the refined keypoints are shown as yellow dots. While the presented patches in this figure have a size of $21\times21$ pixels, the refinement model receives only the $11\times11$ area framed by the red dotted rectangle as input.
• Figure 3: Effect of inaccurate keypoint locations on the accuracy of relative pose estimation. Left: A patch of size $21\times21$ with a true keypoint shown as red dot and yellow dots representing sampled distortions to the keypoint (from top to bottom with a standard deviation of $1$, $2$, and $3$ pixels). The red dotted rectangle shows the $11\times11$ center area of the patch. Right: A graph illustrating the measured AUC5 pose estimation performance on the MegaDepth1500 dataset megadepth, using $2048$ ground truth correspondences perturbed with zero-mean Gaussian noise of varying standard deviation (STD) in pixels.
• Figure 4: Architecture of our attention-guided match refinement. Left: The model takes $11\times11$ image patches $p_{A,i}, p_{B,i}$ (red dotted rectangle) around matched keypoints (red dots) as input. A CNN extracts embeddings $e_{A,i}, e_{B,i}$ which are updated using cross-attention. The score head then maps the updated embeddings $e'_{A,i}, e'_{B,i}$ to score maps $S_{A,i}, S_{B,i}$. A soft-argmax operation on these score maps finally yields the updated keypoint positions (yellow dots). Right: Extension to $n$-view problems. By using one patch as reference $p_{\mathrm{ref}}$ and using a model variant that refines only the second (non-reference) keypoint, consistent refinements can be obtained.
• Figure 5: Example keypoint refinements for XRefine (top two rows), Keypt2Subpx (middle two rows), and PixSfM (bottom two rows). Keypoints are extracted from MegaDepth, using SuperPoint and LightGlue. Each column represents the extracted patches for a given pair of matched keypoints. The same four extracted keypoint matches are refined by the three refinement methods. The presented patches have a size of $21\times21$ pixel, while the $11\times11$ area that is given as input to XRefine and Keypt2Subpx is highlighted by the red dotted square.