GeoRC: A Benchmark for Geolocation Reasoning Chains

TL;DR

The GeoRC paper introduces a first-of-its-kind benchmark for geolocation reasoning chains, collecting 800 expert-written reasoning chains from GeoGuessr challenges to evaluate how well models produce auditable, non-hallucinatory evidence for location predictions. It defines a ground-truth-based grading scheme and investigates three judging approaches (one-to-all LLM, key-points LLM, and VLM-based) to assess candidate chains against expert chains. Experimental results show that, while proprietary VLMs approach human performance in prediction, they lag significantly in generating reliable reasoning traces, with open-weight models performing especially poorly due to hallucinations and misattributions. The work provides open datasets, evaluation protocols, and insights into the limitations of current VLMs, guiding future improvements in fine-grained visual attribute extraction and explainable geolocation reasoning.

Abstract

Vision Language Models (VLMs) are good at recognizing the global location of a photograph -- their geolocation prediction accuracy rivals the best human experts. But many VLMs are startlingly bad at explaining which image evidence led to their prediction, even when their location prediction is correct. The reasoning chains produced by VLMs frequently hallucinate scene attributes to support their location prediction (e.g. phantom writing, imagined infrastructure, misidentified flora). In this paper, we introduce the first benchmark for geolocation reasoning chains. We focus on the global location prediction task in the popular GeoGuessr game which draws from Google Street View spanning more than 100 countries. We collaborate with expert GeoGuessr players, including the reigning world champion, to produce 800 ground truth reasoning chains for 500 query scenes. These expert reasoning chains address hundreds of different discriminative visual attributes such as license plate shape, architecture, and soil properties to name just a few. We evaluate LLM-as-a-judge and VLM-as-a-judge strategies for scoring VLM-generated reasoning chains against our expert reasoning chains and find that Qwen 3 LLM-as-a-judge correlates best with human scoring. Our benchmark reveals that while large, closed-source VLMs such as Gemini and GPT 5 rival human experts at prediction locations, they still lag behind human experts when it comes to producing auditable reasoning chains. Open weights VLMs such as Llama and Qwen catastrophically fail on our benchmark -- they perform only slightly better than a baseline in which an LLM hallucinates a reasoning chain with oracle knowledge of the photo location but no visual information at all. We believe the gap between human experts and VLMs on this task points to VLM limitations at extracting fine-grained visual attributes from high resolution images.

Figures (21)

• Figure 1: GeoRC Benchmark. We curate a dataset of GeoGuessr challenges and associated reasoning chains from three human experts. We then generate reasoning chains from open-weight and proprietary VLMs, and evaluate them through several proposed judging methods.
• Figure 2: Example of Expert Reasoning Chain. Experts typically note discriminative visual features in a coarse-to-fine manner before concluding with a country guess. Expert chains are usually non-exhaustive, only requiring a small number of keypoints to localize the image to a country or region.
• Figure 3: Categories of Geographic Scene Attributes cited by our GeoGuessr experts. Infrastructure and vegetation are the top cited scene attributes.
• Figure 4: Geographical Distribution of GeoRC Dataset that is drawn from popular GeoGuessr world maps inherently conditioned upon Google Streetview coverage.
• Figure 5: One-To-All LLM-as-a-judge Evaluation. One step in the candidate chain is compared to all steps in the expert reasoning chain to compute an F1 score by an LLM judge. An average across all steps in the candidate chain results into the overall score for this candidate.