GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?

TL;DR

GeoGrid-Bench introduces a large-scale, multimodal benchmark to evaluate foundation models on gridded geo-spatial climate data. It leverages ClimRR-derived data across 150 locations and 16 variables to generate 3,200 question–answer pairs via domain-expert templates, presented in tabular and heatmap-based visual formats. The study benchmarks vision-language, language-only, and code-generation-capable models, finding that vision-language models best interpret spatial patterns while code-based approaches struggle with executable analysis, underscoring a need for more agentic capabilities. The benchmark aims to guide practical AI-assisted geo-spatial analysis for climate risk assessment and decision-making, with future work extending regional coverage and data modalities.

Abstract

We present GeoGrid-Bench, a benchmark designed to evaluate the ability of foundation models to understand geo-spatial data in the grid structure. Geo-spatial datasets pose distinct challenges due to their dense numerical values, strong spatial and temporal dependencies, and unique multimodal representations including tabular data, heatmaps, and geographic visualizations. To assess how foundation models can support scientific research in this domain, GeoGrid-Bench features large-scale, real-world data covering 16 climate variables across 150 locations and extended time frames. The benchmark includes approximately 3,200 question-answer pairs, systematically generated from 8 domain expert-curated templates to reflect practical tasks encountered by human scientists. These range from basic queries at a single location and time to complex spatiotemporal comparisons across regions and periods. Our evaluation reveals that vision-language models perform best overall, and we provide a fine-grained analysis of the strengths and limitations of different foundation models in different geo-spatial tasks. This benchmark offers clearer insights into how foundation models can be effectively applied to geo-spatial data analysis and used to support scientific research.

Figures (14)

• Figure 1: Overview of GeoGrid-Bench. The benchmark features questions generated from templates that vary by location, time period, and climate variable, then rewritten with natural language context. Each question is paired with multimodal input—either heatmaps as images or tabular grids of numerical values. We evaluate models on their ability to solve the queries through different modalities—natural language, code, or vision. Ground-truth answers capture find-grained aspects like overall trends, spatial references (from top-left to lower-right), coordinate references (row and column indices), and label references (textual marks on the maps), whenever available.
• Figure 2: We prepare every data sample in one of the four formats: (a) 2D table as a textual string. (b) standalone heatmap; (c) heatmap with overlaid numerical annotations at each grid cell; (d) heatmap overlaid on an actual geographic base map. These formats reflect real-world climate data practices and differ markedly from typical natural images seen by foundation models. More in Appendix \ref{['sec:more_vis']}.
• Figure 3: Overview of the example curation process. Each example in GeoGrid-Bench is constructed by combining a query template with sampled climate variables, locations, and time frames from real-world climate data. Each template is paired with a corresponding oracle code that deterministically generates target answers for all filled-in question instances under that template.
• Figure 4: Evaluation results. The top table shows OpenAI models and the bottom table shows open-source models. Each row corresponds to one model with one data modality—language-only, language and code, or language and vision, while each column represents a query template in Table \ref{['tab:template_questions']}.
• Figure 5: More evaluation results. The top table shows OpenAI models and the bottom table shows open-source models evaluated under different data modalities. Columns represent fine-grained answer aspects defined in Section \ref{['sec:aspect']}, including trend, spatial references, coordinate references, and label references. There exist NaN values since the label reference is only available for the vision modality.