STRMs: Spatial Temporal Reasoning Models for Vision-Based Localization Rivaling GPS Precision

TL;DR

This work tackles centimeter-level vision-based localization in dynamic, large-scale environments by reframing localization as a generative regression task. It introduces Spatial Temporal Reasoning Models (STRMs), specifically VAE-RNN and VAE-Transformer, that map sequences of first-person views to global map perspectives and coordinates without relying on dense satellite image databases. The Transformer-based STRM achieves state-of-the-art localization performance (AUC up to $0.777$) with a lightweight model (~$77$ MB) and real-time inference (~$10.9$ FPS) that rivals smartphone GPS in real-world scenarios, while offering substantial computational efficiency. The results support a cognitive-inspired, regionally specialized deployment strategy for location-specific autonomous driving and point toward future work on temporal robustness and long-term environmental changes.

Abstract

This paper explores vision-based localization through a biologically-inspired approach that mirrors how humans and animals link views or perspectives when navigating their world. We introduce two sequential generative models, VAE-RNN and VAE-Transformer, which transform first-person perspective (FPP) observations into global map perspective (GMP) representations and precise geographical coordinates. Unlike retrieval-based methods, our approach frames localization as a generative task, learning direct mappings between perspectives without relying on dense satellite image databases. We evaluate these models across two real-world environments: a university campus navigated by a Jackal robot and an urban downtown area navigated by a Tesla sedan. The VAE-Transformer achieves impressive precision, with median deviations of 2.29m (1.37% of environment size) and 4.45m (0.35% of environment size) respectively, outperforming both VAE-RNN and prior cross-view geo-localization approaches. Our comprehensive Localization Performance Characteristics (LPC) analysis demonstrates superior performance with the VAE-Transformer achieving an AUC of 0.777 compared to 0.295 for VIGOR 200 and 0.225 for TransGeo, establishing a new state-of-the-art in vision-based localization. In some scenarios, our vision-based system rivals commercial smartphone GPS accuracy (AUC of 0.797) while requiring 5x less GPU memory and delivering 3x faster inference than existing methods in cross-view geo-localization. These results demonstrate that models inspired by biological spatial navigation can effectively memorize complex, dynamic environments and provide precise localization with minimal computational resources.

Figures (6)

• Figure 1: Dataset collection paths for both robotic platforms. Colors indicate RTK-GPS accuracy in meters, with yellow representing higher precision.
• Figure 2: Model architectures for vision-based localization. Both encode FPP images and generate GMP images and coordinate prediction, but differ in sequential processing: VAE-RNN uses RNNs, while VAE-Transformer uses a causal transformer encoder.
• Figure 3: Localization performance characteristics (LPC) comparison on Jackal (top) and Tesla (bottom) datasets. Curves show the percentage of accurate localization measurements within a given meter deviation threshold. The color bands of our models represent confidence band across three runs with different random seeds. AUC values are shown for each method, with higher values indicating better overall performance across all thresholds.
• Figure 4: Localization performance comparison between VAE-RNN (green), VAE-Transformer (blue), and Phone GPS (red) on Jackal (top) and Tesla (bottom) datasets. The plots show deviation from RTK-GPS ground truth in meters over time.
• Figure 5: Localization performance characteristics comparison between models trained with and without reconstruction (w/ Recon vs. w/o Recon) across both datasets. The ablation shows reconstruction's importance increases with environmental complexity, particularly evident in the Tesla urban downtown dataset where performance drops without it.