Learning Affine Correspondences by Integrating Geometric Constraints

TL;DR

The paper addresses the challenge of extracting accurate affine correspondences for robust geometric estimation. It presents a dense, geometry-constrained pipeline that jointly learns dense point matches and local affine transformations, augmented by an affine Sampson-distance loss and a decoupled training scheme. Experimental results on HPatches, MegaDepth, ScanNet, and KITTI show state-of-the-art accuracy for image matching and substantial improvements in relative pose estimation when using affine correspondences. The approach offers a practical pathway to more reliable geometric inference in challenging scenes with large viewpoint changes.

Abstract

Affine correspondences have received significant attention due to their benefits in tasks like image matching and pose estimation. Existing methods for extracting affine correspondences still have many limitations in terms of performance; thus, exploring a new paradigm is crucial. In this paper, we present a new pipeline designed for extracting accurate affine correspondences by integrating dense matching and geometric constraints. Specifically, a novel extraction framework is introduced, with the aid of dense matching and a novel keypoint scale and orientation estimator. For this purpose, we propose loss functions based on geometric constraints, which can effectively improve accuracy by supervising neural networks to learn feature geometry. The experimental show that the accuracy and robustness of our method outperform the existing ones in image matching tasks. To further demonstrate the effectiveness of the proposed method, we applied it to relative pose estimation. Affine correspondences extracted by our method lead to more accurate poses than the baselines on a range of real-world datasets. The code is available at https://github.com/stilcrad/DenseAffine.

Figures (4)

• Figure 1: Image matching with large viewpoint change. Correct matches are green, and incorrect ones red. Our method leads to more correct matches than the LoFTR sun2021loftr and DKM DKM.
• Figure 2: The overview of our method. (a) Abundant accurate point correspondences, encouraged to comply with epipolar constraints through the training loss, are obtained via a dense matching sub-network. (b) The second sub-network is used to estimate the orientation $\mathbf O_i$ and scale $\mathbf S_i$ of each patch and estimate the residual shape $\mathbf A_i"$. (c) Affine correspondences between the two images are estimated.
• Figure 3: The training pipeline. The network starts with training only the dense point matcher supervised by the proposed Sampson distance-based point correspondence loss. This network extracts a dense warp between the images. Next, the point matcher sub-network is frozen, and we train the affine shape extractor to minimize the proposed affine loss, leveraging the epipolar geometry-based constraints.
• Figure 4: The mean matching accuracy (MMA; higher is better) at different thresholds (in pixels) on the HPatches Dataset Hptches.