GeoVLM: Improving Automated Vehicle Geolocalisation Using Vision-Language Matching

TL;DR

Cross-view geolocalisation for automated vehicles often achieves high top-10 recall but struggles to rank the exact top match due to visually similar scenes. GeoVLM introduces a trainable reranking framework that leverages zero-shot vision-language model–generated cross-view language descriptions to disambiguate candidates, combining image and caption embeddings in a shared latent space with a margin-based loss. Evaluations on VIGOR, CVUK, and University-1652 show consistent top-1 and top-5 gains and provide qualitative visualizations of language-driven reasoning, indicating that language descriptions complement visual cues for robust CVGL. While results are promising, offline caption generation and VLM latency present practical challenges for real-time deployment, suggesting future work toward real-time integration on robotic/vehicle platforms.

Abstract

Cross-view geo-localisation identifies coarse geographical position of an automated vehicle by matching a ground-level image to a geo-tagged satellite image from a database. Despite the advancements in Cross-view geo-localisation, significant challenges still persist such as similar looking scenes which makes it challenging to find the correct match as the top match. Existing approaches reach high recall rates but they still fail to rank the correct image as the top match. To address this challenge, this paper proposes GeoVLM, a novel approach which uses the zero-shot capabilities of vision language models to enable cross-view geo-localisation using interpretable cross-view language descriptions. GeoVLM is a trainable reranking approach which improves the best match accuracy of cross-view geo-localisation. GeoVLM is evaluated on standard benchmark VIGOR and University-1652 and also through real-life driving environments using Cross-View United Kingdom, a new benchmark dataset introduced in this paper. The results of the paper show that GeoVLM improves retrieval performance of cross-view geo-localisation compared to the state-of-the-art methods with the help of explainable natural language descriptions. The code is available at https://github.com/CAV-Research-Lab/GeoVLM

Figures (4)

• Figure 1: Overview of proposed GeoVLM pipeline for CVGL. The left side of the figure (blue section) illustrates Phase 1, where the network is frozen and top-k reference images are ranked for each query using pretrained cross-view encoders. The right side (green section) represents Phase 2, the trainable reranking module, where image and text embeddings for each candidate top-k and query image combination are projected into the same latent space and fused to produce a refined ranking score
• Figure 2: Visualization of cross-view images with their corresponding captions and the attention of their text tokens on the relevant part of the image.
• Figure 3: Visualization of a correct match from the CVUK dataset with corresponding scene descriptions. The descriptions were generated using our MCQ-based VQA approach, where BLIP responded to 30 structured questions about the scene.
• Figure 4: Visual clusters of image descriptions for various locations in the CVUK and VIGOR dataset. Each color represents a unique city or state, highlighting the natural clustering of the scene description embeddings in a reduced- dimensional space