MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering

TL;DR

MapIQ tackles the gap in Map-VQA by introducing a diverse benchmark with $14{,}706$ QA pairs across choropleth, cartogram, and proportional symbol maps, spanning six themes and six visual-analytic tasks. It evaluates seven MLLMs under zero-shot prompts and compares results to a human baseline, while testing robustness to 15 map-design variations. The study finds Claude 3.5 Sonnet as the most robust model, with open-source Qwen2-VL competitive in several tasks, yet all MLLMs lag far behind human performance and show theme- and design-dependent biases. These findings highlight the need for more generalized, context-aware geospatial reasoning and motivate future work on broader map types, richer prompting strategies, and improved visual grounding in multimodal models.

Abstract

Recent advancements in multimodal large language models (MLLMs) have driven researchers to explore how well these models read data visualizations, e.g., bar charts, scatter plots. More recently, attention has shifted to visual question answering with maps (Map-VQA). However, Map-VQA research has primarily focused on choropleth maps, which cover only a limited range of thematic categories and visual analytical tasks. To address these gaps, we introduce MapIQ, a benchmark dataset comprising 14,706 question-answer pairs across three map types: choropleth maps, cartograms, and proportional symbol maps spanning topics from six distinct themes (e.g., housing, crime). We evaluate multiple MLLMs using six visual analytical tasks, comparing their performance against one another and a human baseline. An additional experiment examining the impact of map design changes (e.g., altered color schemes, modified legend designs, and removal of map elements) provides insights into the robustness and sensitivity of MLLMs, their reliance on internal geographic knowledge, and potential avenues for improving Map-VQA performance.

Figures (13)

• Figure 1: Three baseline maps. (a) Choropleth, (b) Proportional Symbol, and (c) Cartogram
• Figure 2: Performance of Humans and 7 MLLMs (overall and individual) across 6 Task Types and 3 Map Types. Values represent average performance scores (in %) calculated separately for each task and map type. Bolded values indicate the best-performing model for each experimental condition.
• Figure 3: Performance of Humans and 7 MLLMs (overall and individual) across 4 Question Types and 6 Themes. Values represent average performance scores (in %) computed for each question type and theme. Bolded values indicate the best-performing model for each experimental condition.
• Figure 4: Mean performance differences (in %) between open- and closed-source models across four experimental variables: (a) Task Type, (b) Theme, (c) Question Type, and (d) Map Type. Bars extending to the right of zero indicate better performance by closed-source models, while bars to the left indicate better performance by open-source models.
• Figure 5: Difference in performance of Qwen2-VL relative to the baseline across 15 map design variations, broken down by (a) Task Type and (b) Map Type. Each cell represents the percentage change in accuracy for a specific task or map type under a given variation.