RDD: Robust Feature Detector and Descriptor using Deformable Transformer

TL;DR

This work introduces Robust Deformable Detector (RDD), a two-branch feature detector/descriptor that leverages deformable transformer attention to capture global context and geometric invariance for robust matching under large viewpoint, scale, and illumination changes. By decoupling descriptor and keypoint learning into specialized branches, and employing efficient deformable attention, RDD achieves state-of-the-art performance in sparse and semi-dense matching across standard benchmarks and newly introduced MegaDepth-View and Air-to-Ground datasets. The method is validated through extensive experiments on relative pose estimation, homography estimation, and visual localization, and is supported by ablations and runtime analyses. The introduction of the Air-to-Ground dataset and the MegaDepth-View benchmark provides a more challenging evaluation setting for cross-view reconstruction, highlighting RDD's practical impact for robust 3D vision tasks.

Abstract

As a core step in structure-from-motion and SLAM, robust feature detection and description under challenging scenarios such as significant viewpoint changes remain unresolved despite their ubiquity. While recent works have identified the importance of local features in modeling geometric transformations, these methods fail to learn the visual cues present in long-range relationships. We present Robust Deformable Detector (RDD), a novel and robust keypoint detector/descriptor leveraging the deformable transformer, which captures global context and geometric invariance through deformable self-attention mechanisms. Specifically, we observed that deformable attention focuses on key locations, effectively reducing the search space complexity and modeling the geometric invariance. Furthermore, we collected an Air-to-Ground dataset for training in addition to the standard MegaDepth dataset. Our proposed method outperforms all state-of-the-art keypoint detection/description methods in sparse matching tasks and is also capable of semi-dense matching. To ensure comprehensive evaluation, we introduce two challenging benchmarks: one emphasizing large viewpoint and scale variations, and the other being an Air-to-Ground benchmark -- an evaluation setting that has recently gaining popularity for 3D reconstruction across different altitudes.

Figures (6)

• Figure 2: An overview of our network architecture. Descriptor Branch $\mathcal{F}_D$ and Keypoint Branch $\mathcal{F}_K$ process an input image $I \in \mathbb{R}^{H \times W \times 3}$ independently. Descriptor Branch: 4 layers of multiscale feature maps $\{\mathbf{x}_{res}^l\}_{l=1}^{L}$ are extracted by passing $I$ through ResNet-50 he2015deepresiduallearningimage$\mathcal{F}_{res}$. Additional feature map at scale of $1/64$ is added by applying a simple CNN on the last feature map and then they are feed to a transformer encoder $\mathcal{F}_e$ with positional embeddings vaswani2017attention. We up-sample all feature maps output by $\mathcal{F}_e$ to size $H/k \times W/k$ where $k=4$ is the patch size. Feature maps are then summed together to generate the dense descriptor map $D$. A classification head $\mathcal{F}_m$ is applied to the descriptor map to estimate a matchability map $M$. Keypoint Branch:$I$ passes through a lightweight CNN with residual connection he2015deepresiduallearningimage$\mathcal{F}_{cnn}$ to capture multi-scale features $\{\mathbf{x}_{cnn}^l\}_{l=1}^{L}$. Features are then up-sampled to size $H \times W$ and concatenated to generate a feature map of $H \times W \times 64$. A score map $S$ is estimated by a classification head $\mathcal{F}_s$. Final sub-pixel keypoints are detected using DKD Zhao2022ALIKE.
• Figure 3: Qualitative Results on MegaDepth. RDD* and RDD are qualitatively compared to DeDoDe-V2-G edstedt2024dedodeedstedt2024dedodev2 ALIKED zhao2023aliked and XFeat* potje2024cvpr. RDD* and RDD are more robust compared DeDoDe-G* and ALIKED under challenging scenarios like large scale and viewpoint changes. The red color indicates epipolar error beyond $1\times10^{-4}$ (in the normalized image coordinates)
• Figure 4: Example Pairs from MegaDepth-View and Air-to-Ground The top section shows example pairs from the MegaDepth-View benchmark, which emphasizes large viewpoint shifts and scale differences. The bottom section presents example pairs from the Air-to-Ground dataset/benchmark, designed for the novel task of matching aerial images with ground images.
• Figure 5: More Qualitative Results on MegaDepth-1500. RDD* outperforms DeDoDe-G* in semi-dense matching setting with 30,000 keypoints with a better runtime efficiency \ref{['tab:time_ex']}. The red color indicates epipolar error beyond $1\times10^{-4}$ (in the normalized image coordinates).
• Figure 6: More Qualitative Results on MegaDepth-View. RDD and RDD* are robust under large viewpoint shifts and scale differences. The red color indicates epipolar error beyond $1\times10^{-4}$ (in the normalized image coordinates).