Image-Based Geolocation Using Large Vision-Language Models

TL;DR

This work analyzes the privacy risks of image-based geolocation posed by large vision-language models (LVLMs) and introduces Ethan, a Chain-of-Thought–driven framework that mimics human geoguessing to boost geolocation accuracy. Through a large-scale evaluation on ~50,000 ground-truth samples, Ethan outperforms traditional methods and human benchmarks, achieving high GeoGuessr scores and sub-kilometer precision in many cases. The study also identifies dataset integrity issues and proposes a robust dataset and privacy-preserving strategies, highlighting the need for responsible AI development to safeguard location privacy. Overall, the paper provides a comprehensive assessment of LVLM capabilities in geolocation, introduces practical techniques to enhance performance, and discusses mitigation strategies to protect user privacy in real-world deployments.

Abstract

Geolocation is now a vital aspect of modern life, offering numerous benefits but also presenting serious privacy concerns. The advent of large vision-language models (LVLMs) with advanced image-processing capabilities introduces new risks, as these models can inadvertently reveal sensitive geolocation information. This paper presents the first in-depth study analyzing the challenges posed by traditional deep learning and LVLM-based geolocation methods. Our findings reveal that LVLMs can accurately determine geolocations from images, even without explicit geographic training. To address these challenges, we introduce \tool{}, an innovative framework that significantly enhances image-based geolocation accuracy. \tool{} employs a systematic chain-of-thought (CoT) approach, mimicking human geoguessing strategies by carefully analyzing visual and contextual cues such as vehicle types, architectural styles, natural landscapes, and cultural elements. Extensive testing on a dataset of 50,000 ground-truth data points shows that \tool{} outperforms both traditional models and human benchmarks in accuracy. It achieves an impressive average score of 4550.5 in the GeoGuessr game, with an 85.37\% win rate, and delivers highly precise geolocation predictions, with the closest distances as accurate as 0.3 km. Furthermore, our study highlights issues related to dataset integrity, leading to the creation of a more robust dataset and a refined framework that leverages LVLMs' cognitive capabilities to improve geolocation precision. These findings underscore \tool{}'s superior ability to interpret complex visual data, the urgent need to address emerging security vulnerabilities posed by LVLMs, and the importance of responsible AI development to ensure user privacy protection.

Figures (6)

• Figure 1: Common scenarios of how adversaries extract private geolocation information from the victims.
• Figure 2: Overview of our work
• Figure 3: Threat Model for Geolocation Privacy using Large Vision Language Models (LVLMs)
• Figure 4: Visual representation of image localizability spectrum, categorized from non-localizable scenes to recognizable landmarks, illustrating the diversity in the dataset.
• Figure 5: Categorization of images in the original dataset based on their localizability: "Minimal Context" for images with minimal geographic markers, "Contextually Ambiguous" for visually descriptive but non-localizable images, and "Highly Misleading" for ambiguous images leading to significant localization errors.