Evaluating Precise Geolocation Inference Capabilities of Vision Language Models

TL;DR

This work demonstrates that modern Vision-Language Models can infer precise geographic coordinates from unseen Street View imagery, challenging assumptions about privacy in multimodal AI. By constructing a global Street View–based dataset of 1602 images and evaluating base VLMs on single-image geolocation with GeoGuessr-like prompts, the authors show median errors can fall well below 300 km for many models. They further show that lightweight VLM agents, equipped with Street View look-around and Google Lens tools, can reduce mean geolocation error by up to about 30% for large models and even outperform top GeoGuessr players in a few guesses. The study highlights significant privacy implications of accessible VLM capabilities, discusses limitations in geographic coverage, and proposes future work on more advanced agents and anonymization to mitigate risks.

Abstract

The prevalence of Vision-Language Models (VLMs) raises important questions about privacy in an era where visual information is increasingly available. While foundation VLMs demonstrate broad knowledge and learned capabilities, we specifically investigate their ability to infer geographic location from previously unseen image data. This paper introduces a benchmark dataset collected from Google Street View that represents its global distribution of coverage. Foundation models are evaluated on single-image geolocation inference, with many achieving median distance errors of <300 km. We further evaluate VLM "agents" with access to supplemental tools, observing up to a 30.6% decrease in distance error. Our findings establish that modern foundation VLMs can act as powerful image geolocation tools, without being specifically trained for this task. When coupled with increasing accessibility of these models, our findings have greater implications for online privacy. We discuss these risks, as well as future work in this area.

Figures (7)

• Figure 1: A snapshot of Google Street View images in the full benchmark dataset.
• Figure 2: Overview of distance error for select benchmarked models, with human benchmark for reference. Human benchmark mean is 5633 km. Full results can be found in Appendix B.
• Figure 3: Median distance error for "VLM + Street View" agents over 5 guesses. Larger models typically improve in performance.
• Figure 4: Median distance error for "VLM + Street View" agents over 5 guesses. Smaller models achieved less improvement in performance.
• Figure 5: "VLM + Street View" agent compared to median error for competitive GeoGuessr players and specialized architecture, PIGEON haas2024pigeonpredictingimagegeolocations. In both human baselines, the entire photosphere is given.