Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPSBench: Do Large Language Models Understand GPS Coordinates?

Thinh Hung Truong, Jey Han Lau, Jianzhong Qi

TL;DR

GPSBench systematically probes intrinsic geospatial reasoning in LLMs across 17 tasks and two tracks (Pure GPS and Applied) by deriving ground truth from geodetic computations and GeoNames data. The benchmark reveals that LLMs handle basic coordinate operations reasonably, but struggle with complex spherical geometry and fine-grained place localization, with geographic knowledge showing hierarchical degradation from country to city level. GPS augmentation improves downstream spatial tasks, while finetuning can boost geometric computation at the expense of world knowledge. Model scaling generally boosts performance, and robustness to coordinate noise indicates genuine geographic understanding rather than memorization. The work provides a publicly available dataset and code to guide future model development and evaluation for location-aware AI systems.

Abstract

Large Language Models (LLMs) are increasingly deployed in applications that interact with the physical world, such as navigation, robotics, or mapping, making robust geospatial reasoning a critical capability. Despite that, LLMs' ability to reason about GPS coordinates and real-world geography remains underexplored. We introduce GPSBench, a dataset of 57,800 samples across 17 tasks for evaluating geospatial reasoning in LLMs, spanning geometric coordinate operations (e.g., distance and bearing computation) and reasoning that integrates coordinates with world knowledge. Focusing on intrinsic model capabilities rather than tool use, we evaluate 14 state-of-the-art LLMs and find that GPS reasoning remains challenging, with substantial variation across tasks: models are generally more reliable at real-world geographic reasoning than at geometric computations. Geographic knowledge degrades hierarchically, with strong country-level performance but weak city-level localization, while robustness to coordinate noise suggests genuine coordinate understanding rather than memorization. We further show that GPS-coordinate augmentation can improve in downstream geospatial tasks, and that finetuning induces trade-offs between gains in geometric computation and degradation in world knowledge. Our dataset and reproducible code are available at https://github.com/joey234/gpsbench

GPSBench: Do Large Language Models Understand GPS Coordinates?

TL;DR

GPSBench systematically probes intrinsic geospatial reasoning in LLMs across 17 tasks and two tracks (Pure GPS and Applied) by deriving ground truth from geodetic computations and GeoNames data. The benchmark reveals that LLMs handle basic coordinate operations reasonably, but struggle with complex spherical geometry and fine-grained place localization, with geographic knowledge showing hierarchical degradation from country to city level. GPS augmentation improves downstream spatial tasks, while finetuning can boost geometric computation at the expense of world knowledge. Model scaling generally boosts performance, and robustness to coordinate noise indicates genuine geographic understanding rather than memorization. The work provides a publicly available dataset and code to guide future model development and evaluation for location-aware AI systems.

Abstract

Large Language Models (LLMs) are increasingly deployed in applications that interact with the physical world, such as navigation, robotics, or mapping, making robust geospatial reasoning a critical capability. Despite that, LLMs' ability to reason about GPS coordinates and real-world geography remains underexplored. We introduce GPSBench, a dataset of 57,800 samples across 17 tasks for evaluating geospatial reasoning in LLMs, spanning geometric coordinate operations (e.g., distance and bearing computation) and reasoning that integrates coordinates with world knowledge. Focusing on intrinsic model capabilities rather than tool use, we evaluate 14 state-of-the-art LLMs and find that GPS reasoning remains challenging, with substantial variation across tasks: models are generally more reliable at real-world geographic reasoning than at geometric computations. Geographic knowledge degrades hierarchically, with strong country-level performance but weak city-level localization, while robustness to coordinate noise suggests genuine coordinate understanding rather than memorization. We further show that GPS-coordinate augmentation can improve in downstream geospatial tasks, and that finetuning induces trade-offs between gains in geometric computation and degradation in world knowledge. Our dataset and reproducible code are available at https://github.com/joey234/gpsbench
Paper Structure (107 sections, 9 equations, 11 figures, 13 tables)

This paper contains 107 sections, 9 equations, 11 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Spatial Cognition Framework
  4. Geospatial Benchmarks
  5. Geographic knowledge and bias
  6. Spatial reasoning
  7. Tool-use
  8. GPSBench
  9. Benchmark Design
  10. Data Generation
  11. Data source
  12. Sample generation pipeline
  13. Ground truth computation
  14. Experiments
  15. Experiment setting
  16. ...and 92 more sections

Figures (11)

  • Figure 1: Geographic coverage of GPSBench. 18,196 unique locations from GeoNames span six continents: Asia (33.7%), Europe (23.2%), North America (16.8%), South America (13.9%), Africa (9.4%), and Oceania (2.7%).
  • Figure 2: Applied vs Pure GPS Track performance. Most models cluster below the diagonal (favoring Applied), while GPT-5.1 and Gemini-2.5-Pro uniquely excel at Pure GPS computation.
  • Figure 3: Mean accuracy across all models per task, sorted by difficulty. Error bars show $\pm$1 standard deviation. Tasks cluster into solved ($>$95%), brittle (25--95%), and unsolved ($<$25%) tiers.
  • Figure 4: Regional performance by subregion across all GPSBench tasks. Applied Track shows the largest gaps: North America (75.8%) vs. East Asia (59.6%), a 16.2% difference. Pure GPS Track is more uniform across most subregions.
  • Figure 5: Hierarchical degradation in geographic knowledge. Models perform well at country-level but accuracy drops sharply for finer granularities, with city-level accuracy below 25% for all.
  • ...and 6 more figures