Multiple-environment Self-adaptive Network for Aerial-view Geo-localization

TL;DR

This paper tackles the challenge of cross-view aerial geo-localization under weather-induced domain shifts by introducing MuSe-Net, a two-branch network that separately learns environment-specific style information and self-adapts content features via Residual SPADE. By forcing explicit style discrimination and dynamically modulating feature activations, MuSe-Net reduces environmental gaps while preserving identity information, achieving superior results on University-1652, SUES-200, and CVUSA and showing promise under unseen extreme weather. The approach advances domain generalization for geo-localization and offers practical robustness for drone navigation and safety in varied weather conditions. Overall, MuSe-Net provides a scalable, end-to-end solution for robust cross-view localization in multi-environment scenarios, with potential implications for real-world drone operations.

Abstract

Aerial-view geo-localization tends to determine an unknown position through matching the drone-view image with the geo-tagged satellite-view image. This task is mostly regarded as an image retrieval problem. The key underpinning this task is to design a series of deep neural networks to learn discriminative image descriptors. However, existing methods meet large performance drops under realistic weather, such as rain and fog, since they do not take the domain shift between the training data and multiple test environments into consideration. To minor this domain gap, we propose a Multiple-environment Self-adaptive Network (MuSe-Net) to dynamically adjust the domain shift caused by environmental changing. In particular, MuSe-Net employs a two-branch neural network containing one multiple-environment style extraction network and one self-adaptive feature extraction network. As the name implies, the multiple-environment style extraction network is to extract the environment-related style information, while the self-adaptive feature extraction network utilizes an adaptive modulation module to dynamically minimize the environment-related style gap. Extensive experiments on two widely-used benchmarks, i.e., University-1652 and CVUSA, demonstrate that the proposed MuSe-Net achieves a competitive result for geo-localization in multiple environments. Furthermore, we observe that the proposed method also shows great potential to the unseen extreme weather, such as mixing the fog, rain and snow.

Figures (4)

• Figure 1: Examples of synthesized environments on University-1652 zheng_university-1652_nodate, which raise challenges on the robustness of the drone vision. Specifically, we generate these images by adding different environmental styles into the normal drone-view images. Each column and row corresponds to different geographical locations and environments.
• Figure 2: (I) A schematic overview of MuSe-Net. One batch of inputs contains the same number of satellite and drone images, and the style of satellite images is invariable. MuSe-Net consists of two branches. Purple blocks indicate the multiple-environment style extraction branch (purple branch), and pink blocks denote the self-adaptive feature extraction branch (pink branch). The purple branch is employed to group the same style features. Then style features extracted from the style encoder are fed into Residual SPADE. Residual SPADE is embedded into the content encoder, which belongs to the pink branch. The pink branch is applied to narrow down the distance of inputs with the same geo-tag. (II) Detailed demonstration of information interaction between the style encoder and the content encoder. The extracted style information is first convolved to produce the modulation parameters in Residual SPADE. After that, we utilize the learned modulation parameters to modulate middle features of inputs in the content encoder. (III) Illustration of the location of Residual SPADE in one bottleneck of the content encoder (a) and the calculation flow of the modulation operation (b). In Residual SPADE, a group of modulation parameters $w$ and $b$ comes from two convolutional layers, i.e., Conv_w1 and Conv_b1. Afterwards the learned $w$ and $b$ are applied to modulate the activation of instance normalization (IN).
• Figure 3: Examples of a street-view panorama and its synthesized environments on CVUSA zhai_predicting_2017.
• Figure 4: Visualization of heatmaps generated by our method and Top-5 retrieval results for a drone-view image in different conditions. The true matches are in yellow boxes, and the false matches are displayed in blue boxes.