Weakly-supervised Camera Localization by Ground-to-satellite Image Registration

TL;DR

A weakly supervised learning strategy for ground-to-satellite image registration when only noisy pose labels for ground images are available for network training that derives positive and negative satellite images for each ground image and leverages contrastive learning to learn feature representations for ground and satellite images useful for translation estimation.

Abstract

The ground-to-satellite image matching/retrieval was initially proposed for city-scale ground camera localization. This work addresses the problem of improving camera pose accuracy by ground-to-satellite image matching after a coarse location and orientation have been obtained, either from the city-scale retrieval or from consumer-level GPS and compass sensors. Existing learning-based methods for solving this task require accurate GPS labels of ground images for network training. However, obtaining such accurate GPS labels is difficult, often requiring an expensive {\color{black}Real Time Kinematics (RTK)} setup and suffering from signal occlusion, multi-path signal disruptions, \etc. To alleviate this issue, this paper proposes a weakly supervised learning strategy for ground-to-satellite image registration when only noisy pose labels for ground images are available for network training. It derives positive and negative satellite images for each ground image and leverages contrastive learning to learn feature representations for ground and satellite images useful for translation estimation. We also propose a self-supervision strategy for cross-view image relative rotation estimation, which trains the network by creating pseudo query and reference image pairs. Experimental results show that our weakly supervised learning strategy achieves the best performance on cross-area evaluation compared to recent state-of-the-art methods that are reliant on accurate pose labels for supervision.

Figures (8)

• Figure 1: We derive positive and negative satellite images for each ground-view image based on its coarse location information. A similarity map between the ground view and each satellite image is computed when aligned at different locations. Our training objective aims to maximize the maximum similarity in the positive similarity map while minimizing the maximum similarity in the negative similarity map.
• Figure 2: Self-supervised rotation estimator.
• Figure 3: Translation estimation framework. During training, the weights of the rotation estimator are fixed, and we only train the two-branch feature extractors. Other uncolored blocks indicate no trainable parameters are involved.
• Figure 4: Visualization of learned confidence maps for ground images and estimated poses by our method.
• Figure 5: (a) Ground image; (b) Synthesized overhead view feature map from the ground image; (c) Satellite image feature map; (d) Estimated location probability map and camera orientation indicated by the red arrow.