Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data

Chenhui Zhang, Sherrie Wang

TL;DR

This paper presents an Earth Observation–focused benchmark to evaluate instruction-following Vision-Language Models on tasks requiring scene understanding, localization/counting, and change detection. By comparing GPT-4V with open VLMs across landmark recognition, RSICD captioning, LULC classification, object localization, counting, and change detection datasets, the study highlights a gap: strong high-level scene understanding does not translate into precise spatial reasoning or temporal change analysis. The findings show GPT-4V excels in open-ended understanding and captioning but struggles with bounding-box localization, accurate counting of small objects, and disaster-change assessment, suggesting that current VLMs need targeted EO data, architecture adjustments, and improved alignment for these fine-grained tasks. The work proposes a publicly available benchmark interface and outlines real-world EO applications where improved VLMs could impact urban monitoring, conservation, and disaster response, while outlining concrete directions for future research including segmentation capabilities and dynamic benchmark updates.

Abstract

Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose a comprehensive benchmark to gauge the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our benchmark includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks. Our benchmark will be made publicly available at https://vleo.danielz.ch/ and on Hugging Face at https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70 for easy model evaluation.

Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data

TL;DR

This paper presents an Earth Observation–focused benchmark to evaluate instruction-following Vision-Language Models on tasks requiring scene understanding, localization/counting, and change detection. By comparing GPT-4V with open VLMs across landmark recognition, RSICD captioning, LULC classification, object localization, counting, and change detection datasets, the study highlights a gap: strong high-level scene understanding does not translate into precise spatial reasoning or temporal change analysis. The findings show GPT-4V excels in open-ended understanding and captioning but struggles with bounding-box localization, accurate counting of small objects, and disaster-change assessment, suggesting that current VLMs need targeted EO data, architecture adjustments, and improved alignment for these fine-grained tasks. The work proposes a publicly available benchmark interface and outlines real-world EO applications where improved VLMs could impact urban monitoring, conservation, and disaster response, while outlining concrete directions for future research including segmentation capabilities and dynamic benchmark updates.

Abstract

Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose a comprehensive benchmark to gauge the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our benchmark includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks. Our benchmark will be made publicly available at https://vleo.danielz.ch/ and on Hugging Face at https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70 for easy model evaluation.
Paper Structure (69 sections, 3 equations, 47 figures, 30 tables)

This paper contains 69 sections, 3 equations, 47 figures, 30 tables.

Table of Contents

  1. Introduction
  2. Desired Capabilities for EO Data.
  3. Model Selection.
  4. Empirical Findings.
  5. Recommended Usage.
  6. Limitations & Future Work.
  7. Scene Understanding
  8. Location Recognition
  9. Goals.
  10. Dataset Construction.
  11. System and Task Prompts.
  12. Evaluation Setup.
  13. Results.
  14. Image Captioning
  15. Goals.
  16. ...and 54 more sections

Figures (47)

  • Figure 1: Task taxonomy for evaluating Vision-Language Models (VLMs) on Earth observation (EO) data. Tasks are organized into boxes by capability --- scene understanding, localization & counting, and change detection --- and top to bottom by image spatial resolution.
  • Figure 2: Examples of inputs and outputs from different benchmark tasks and performance across the 5 VLMs we assess. We only select part of the user prompt and model response for illustration purposes.
  • Figure 3: GPT-4V has scene understanding abilities but cannot accurately count or localize objects. We only select part of the user prompt and model response for illustration purposes.
  • Figure 4: System prompt for location recognition.
  • Figure 5: Example GPT-4V prompt and response for location recognition.
  • ...and 42 more figures