VSFormer: Visual-Spatial Fusion Transformer for Correspondence Pruning

TL;DR

Two-view correspondence pruning is hindered by high outlier ratios and varying scene conditions. VSFormer introduces a Visual-Spatial Fusion Transformer that injects scene visual cues into correspondences, followed by a ContextFormer that jointly models local and global context to refine inlier probabilities and pose estimates. The approach combines a Visual Cues Extractor, a VSFusion module for cross-modal fusion, and a graph-augmented transformer to enhance context modeling, optimized by a hybrid loss with essential-matrix regression. Empirical results on outdoor YFCC100M and indoor SUN3D show consistent improvements over state-of-the-art methods, validating the effectiveness of scene-aware priors and context fusion for robust two-view geometry estimation.

Abstract

Correspondence pruning aims to find correct matches (inliers) from an initial set of putative correspondences, which is a fundamental task for many applications. The process of finding is challenging, given the varying inlier ratios between scenes/image pairs due to significant visual differences. However, the performance of the existing methods is usually limited by the problem of lacking visual cues (\eg texture, illumination, structure) of scenes. In this paper, we propose a Visual-Spatial Fusion Transformer (VSFormer) to identify inliers and recover camera poses accurately. Firstly, we obtain highly abstract visual cues of a scene with the cross attention between local features of two-view images. Then, we model these visual cues and correspondences by a joint visual-spatial fusion module, simultaneously embedding visual cues into correspondences for pruning. Additionally, to mine the consistency of correspondences, we also design a novel module that combines the KNN-based graph and the transformer, effectively capturing both local and global contexts. Extensive experiments have demonstrated that the proposed VSFormer outperforms state-of-the-art methods on outdoor and indoor benchmarks. Our code is provided at the following repository: https://github.com/sugar-fly/VSFormer.

Figures (5)

• Figure 1: Comparison between previous methods and ours. Top: the architecture of previous methods, which lack the visual perception of a scene. Bottom: the architecture of our VSFormer introduces visual cues of a scene to guide correspondence pruning. For visualization purposes, the correspondences (4D) across two-view images are projected into a 2D space by t-SNE Van2008. The circle CA represents the cross-attention layer.
• Figure 2: The architecture of our VSFormer mainly contains Visual Cues Extractor (VCExtractor), Visual-Spatial Fusion (VSFusion) Module, and Context Transformer (ContextFormer). Note that we omit the inlier predictor after ContextFormer for simplicity.
• Figure 3: Illustration of our proposed ContextFormer. Meanwhile, we design a novel graph attention block to mine potential relationships along three different dimensions.
• Figure 4: Partial typical visualization results on two challenging datasets, i.e., YFCC100M, SUN3D. From left to right: the results of OANet++, CLNet, and our VSFormer. From top to bottom: the top three results come from unknown outdoor scenes and the rest come from unknown indoor scenes. The correspondence is drawn in green if it represents the true-positive and red for the false-positive. Best viewed in color.
• Figure 5: Qualitative comparison with/without scene prior.