Efficient Spike-driven Transformer for High-performance Drone-View Geo-Localization

TL;DR

This work tackles DVGL under stringent energy constraints by introducing SpikeViMFormer, a spike-driven transformer framework that remains competitive with ANN-based approaches while dramatically reducing inference energy. It combines a lightweight dual-stream SNN transformer backbone with SSA for selective feature enhancement and SHS for long-range dependency modeling, supervised by HRAL during training. The approach yields strong cross-view retrieval performance with significantly fewer parameters and lower energy consumption, and demonstrates robustness across backbones and extreme sparsity scenarios. These results highlight the practical viability of neuromorphic, spike-driven models for resource-limited DVGL deployments and hardware-friendly deployment prospects.

Abstract

Traditional drone-view geo-localization (DVGL) methods based on artificial neural networks (ANNs) have achieved remarkable performance. However, ANNs rely on dense computation, which results in high power consumption. In contrast, spiking neural networks (SNNs), which benefit from spike-driven computation, inherently provide low power consumption. Regrettably, the potential of SNNs for DVGL has yet to be thoroughly investigated. Meanwhile, the inherent sparsity of spike-driven computation for representation learning scenarios also results in loss of critical information and difficulties in learning long-range dependencies when aligning heterogeneous visual data sources. To address these, we propose SpikeViMFormer, the first SNN framework designed for DVGL. In this framework, a lightweight spike-driven transformer backbone is adopted to extract coarse-grained features. To mitigate the loss of critical information, the spike-driven selective attention (SSA) block is designed, which uses a spike-driven gating mechanism to achieve selective feature enhancement and highlight discriminative regions. Furthermore, a spike-driven hybrid state space (SHS) block is introduced to learn long-range dependencies using a hybrid state space. Moreover, only the backbone is utilized during the inference stage to reduce computational cost. To ensure backbone effectiveness, a novel hierarchical re-ranking alignment learning (HRAL) strategy is proposed. It refines features via neighborhood re-ranking and maintains cross-batch consistency to directly optimize the backbone. Experimental results demonstrate that SpikeViMFormer outperforms state-of-the-art SNNs. Compared with advanced ANNs, it also achieves competitive performance.Our code is available at https://github.com/ISChenawei/SpikeViMFormer

Figures (8)

• Figure 1: DVGL with ANNs vs. SNNs. (a) In ANN-based DVGL, neurons employ real-valued activations and dense MAC operations, which results in high activation density in convolution and attention layers. Such dense computation is energy-intensive, particularly on resource-constrained drone platforms, which leads to rapid battery depletion. (b) In contrast, SNN-based DVGL adopts an event-driven paradigm where neurons emit binary spikes only upon reaching the firing threshold. This replaces MAC with low-power AC operations and leverages sparse activations, thus achieving remarkable energy efficiency.
• Figure 2: Classification vs. DVGL with SNNs. (a) In classification, SNNs only need to map an entire image to a class label, relying mainly on global discriminative cues such as textures or contours. Information compression and sparsity thus have little impact. (b) In DVGL, drone images from oblique viewpoints must be aligned with satellite images from top-down views. This cross-view scenario demands precise alignment. However, spike-induced sparsity amplifies alignment errors, making SNNs prone to confusion when visually similar but semantically different regions (e.g., rooftops with the same color) appear. Therefore, it is essential to capture and model long-range dependencies, as global contextual information helps disambiguate visually similar but semantically different regions and thus mitigates alignment errors.
• Figure 3: Pipeline Overview. The proposed SpikeViMFormer first adopts a dual-stream lightweight spike-driven transformer backbone with shared weights to extract coarse-grained features. On this basis, two auxiliary blocks are employed, including the SSA block and the SHS block. The SSA block is designed to selectively enhance and focus attention on discriminative regions. In addition, it leverages a cosine embedding loss to guide the narrowing of the representational gap between drone-view and satellite-view. Meanwhile, the SHS block is employed to capture long-range dependencies by leveraging hybrid state space dynamics, and it is optimized with a cross-entropy loss to improve semantic consistency and enhance cross-view alignment. In addition, the hierarchical re-ranking alignment strategy optimizes the backbone by enforcing cross-batch consistency, which ensures its effectiveness when auxiliary blocks are discarded at inference stage.
• Figure 4: SSA Block and SHS Block. Illustration of (left) spike-driven selective attention (SSA) block and (right) spike-driven hybrid state space (SHS) block.
• Figure 5: RandomKeepPatches strategy. (a) The input image during inference stage with spatial resolution $H \times W$. (b) A set of patches is randomly sampled from the input image. (c) The final input of the sparse activation is constructed by randomly placing $P$ patches, each with an area of $\tfrac{H \times W \times R}{P}$, where $R$ denotes the predefined retention ratio of the total image area.