DeepDetect: Learning All-in-One Dense Keypoints

TL;DR

DeepDetect tackles the need for dense, semantically meaningful keypoints for reliable image matching and 3D reconstruction in challenging scenes. It introduces a dense detector that unifies seven classical keypoint detectors and two edge detectors through fused supervision masks to train a lightweight ESPNet. The method achieves state-of-the-art performance on the Oxford dataset with high keypoint density, repeatability, and correct matches, as evidenced by density 0.5143, repeatability 0.9582, and 59,003 matches. Its compact footprint (~1.82 MB) enables edge-device deployment while maintaining robustness to photometric and geometric changes, making it suitable for dense SLAM, AR/VR, and autonomous navigation.

Abstract

Keypoint detection is the foundation of many computer vision tasks, including image registration, structure-from motion, 3D reconstruction, visual odometry, and SLAM. Traditional detectors (SIFT, SURF, ORB, BRISK, etc.) and learning based methods (SuperPoint, R2D2, LF-Net, D2-Net, etc.) have shown strong performance yet suffer from key limitations: sensitivity to photometric changes, low keypoint density and repeatability, limited adaptability to challenging scenes, and lack of semantic understanding, often failing to prioritize visually important regions. We present DeepDetect, an intelligent, all-in-one, dense keypoint detector that unifies the strengths of classical detectors using deep learning. Firstly, we create ground-truth masks by fusing outputs of 7 keypoint and 2 edge detectors, extracting diverse visual cues from corners and blobs to prominent edges and textures in the images. Afterwards, a lightweight and efficient model: ESPNet, is trained using these masks as labels, enabling DeepDetect to focus semantically on images while producing highly dense keypoints, that are adaptable to diverse and visually degraded conditions. Evaluations on the Oxford Affine Covariant Regions dataset demonstrate that DeepDetect surpasses other detectors in keypoint density, repeatability, and the number of correct matches, achieving maximum values of 0.5143 (average keypoint density), 0.9582 (average repeatability), and 59,003 (correct matches).

Figures (30)

• Figure 1: Average repeatability of keypoint detectors on Oxford Dataset ref_30.
• Figure 2: Keypoint detection on two images using SIFT with normal (default) thresholds, SIFT with extremely low thresholds, and DeepDetect. Number of detected keypoints: Top Scene --- 279, 23990, 46309 and Bottom Scene --- 1009, 75465, 51115, respectively. SIFT with standard thresholds yields limited keypoints, whereas SIFT with extremely low thresholds produces high number of keypoints, detecting noisy features on semantically irrelevant regions. DeepDetect produces highly dense keypoints with substantially lower noise, well concentrated in semantically important regions of the images.
• Figure 3: Main stages in the development of DeepDetect: (a) Fusion masks are created by using 7 keypoint detectors (SIFT, ORB, BRISK, FAST, AGAST, Harris, Shi-Tomasi) and 2 edge detectors (Canny, Sobel) in the images, combining their strengths to capture rich and diverse visual representations. (b) The fused binary masks are then used as labels for the corresponding images to supervise and train ESPNet (a lightweight, efficient model). (c) DeepDetect is ready to detect all-in-one, dense keypoints with semantic awareness and adaptability to complex image degradations, without requiring manual tuning.
• Figure 4: Binary masks obtained from 7 keypoint and 2 edge detectors, along with their combined version (which provides richer representations for training).
• Figure 5: Training and validation loss curves of DeepDetect.