m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoning

TL;DR

m2sv introduces a scalable, real-world benchmark for map-to-street-view spatial reasoning, isolating the core problem of orienting egocentric Street View imagery to a north-up overhead map. By releasing m2sv-20k and m2sv-sft-11k, the work enables large-scale evaluation and targeted fine-tuning while providing a controlled framework to quantify geometric difficulty through structural cues and human effort. Empirical results show a large gap between human performance ($95\%$) and current vision–language models (best zero-shot $65.2\%$), with adaptation providing modest gains but limited cross-benchmark transfer. Through thorough failure-mode analyses and difficulty measures, the paper highlights systematic gaps in geometric alignment, cue aggregation, and adaptive reasoning, underscoring the need for grounded spatial reasoning and uncertainty-aware inference in future models.

Abstract

Vision--language models (VLMs) achieve strong performance on many multimodal benchmarks but remain brittle on spatial reasoning tasks that require aligning abstract overhead representations with egocentric views. We introduce m2sv, a scalable benchmark for map-to-street-view spatial reasoning that asks models to infer camera viewing direction by aligning a north-up overhead map with a Street View image captured at the same real-world intersection. We release m2sv-20k, a geographically diverse benchmark with controlled ambiguity, along with m2sv-sft-11k, a curated set of structured reasoning traces for supervised fine-tuning. Despite strong performance on existing multimodal benchmarks, the best evaluated VLM achieves only 65.2% accuracy on m2sv, far below the human baseline of 95%. While supervised fine-tuning and reinforcement learning yield consistent gains, cross-benchmark evaluations reveal limited transfer. Beyond aggregate accuracy, we systematically analyze difficulty in map-to-street-view reasoning using both structural signals and human effort, and conduct an extensive failure analysis of adapted open models. Our findings highlight persistent gaps in geometric alignment, evidence aggregation, and reasoning consistency, motivating future work on grounded spatial reasoning across viewpoints.

Figures (8)

• Figure 1: Illustrative m2sv example with paired images and a model-generated reasoning trace. The model correctly infers the viewing direction by aligning stable map geometry with the Street View image.
• Figure 2: Chance-normalized accuracy vs. structural difficulty. Accuracy normalized by chance level as a function of the number of candidate directions (#options). Qwen3-VL-8B (SFT+RL) is evaluated on the full 10k validation set, while Gemini 3 Pro and GPT-5 are evaluated on a 1k subset. Three-option intersections yield the highest normalized gains across models, reflecting the benefit of geometric asymmetry. Error bars denote 95% binomial confidence intervals.
• Figure 3: Accuracy vs. road-azimuth symmetry. Model accuracy as a function of azimuth symmetry quantiles (Q1 = most symmetric, Q5 = most asymmetric). Structural symmetry alone does not predict difficulty: accuracy peaks at intermediate symmetry levels and degrades in both highly symmetric and highly asymmetric regimes. Error bars denote 95% binomial confidence intervals.
• Figure 4: Human response time vs. road-azimuth symmetry. Each point denotes a single example from the 200-example analysis subset. Highly symmetric intersections exhibit a wide range of difficulty, while asymmetric intersections are uniformly easy for humans.
• Figure 5: Accuracy vs. human-perceived difficulty. Accuracy as a function of difficulty defined by human response time. Examples are partitioned into Easy, Medium, and Hard buckets by response-time tertiles. Error bars indicate 95% binomial confidence intervals.