Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing

Madeline Anderson, Miriam Cha, William T. Freeman, J. Taylor Perron, Nathaniel Maidel, Kerri Cahoy

TL;DR

The paper addresses the lack of paired image-text data in remote sensing by leveraging maps and metadata to generate detailed captions via a multimodal LLM, culminating in the fMoW-mm dataset. It introduces a hallucination metric, FDR, defined as $FDR = 1 - \frac{\sum_{c \in C} \mathds{1}_{R}(c)}{K}$, to quantify false-positive proper-noun mentions and demonstrates that map-enhanced captions reduce such hallucinations. Through ablations and few-shot evaluations on the DIOR dataset, the authors show that fMoW-mm yields superior performance for automatic target recognition compared to existing vision-language remote sensing datasets. Overall, the work demonstrates that external contextual data can substantially improve caption quality and downstream few-shot tasks in data-scarce remote sensing settings.

Abstract

Vision language models have achieved impressive results across various fields. However, adoption in remote sensing remains limited, largely due to the scarcity of paired image-text data. To bridge this gap, synthetic caption generation has gained interest, traditionally relying on rule-based methods that use metadata or bounding boxes. While these approaches provide some description, they often lack the depth needed to capture complex wide-area scenes. Large language models (LLMs) offer a promising alternative for generating more descriptive captions, yet they can produce generic outputs and are prone to hallucination. In this paper, we propose a new method to enhance vision-language datasets for remote sensing by integrating maps as external data sources, enabling the generation of detailed, context-rich captions. Additionally, we present methods to measure and mitigate hallucinations in LLM-generated text. We introduce fMoW-mm, a multimodal dataset incorporating satellite imagery, maps, metadata, and text annotations. We demonstrate its effectiveness for automatic target recognition in few-shot settings, achieving superior performance compared to other vision-language remote sensing datasets.

Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing

TL;DR

The paper addresses the lack of paired image-text data in remote sensing by leveraging maps and metadata to generate detailed captions via a multimodal LLM, culminating in the fMoW-mm dataset. It introduces a hallucination metric, FDR, defined as , to quantify false-positive proper-noun mentions and demonstrates that map-enhanced captions reduce such hallucinations. Through ablations and few-shot evaluations on the DIOR dataset, the authors show that fMoW-mm yields superior performance for automatic target recognition compared to existing vision-language remote sensing datasets. Overall, the work demonstrates that external contextual data can substantially improve caption quality and downstream few-shot tasks in data-scarce remote sensing settings.

Abstract

Vision language models have achieved impressive results across various fields. However, adoption in remote sensing remains limited, largely due to the scarcity of paired image-text data. To bridge this gap, synthetic caption generation has gained interest, traditionally relying on rule-based methods that use metadata or bounding boxes. While these approaches provide some description, they often lack the depth needed to capture complex wide-area scenes. Large language models (LLMs) offer a promising alternative for generating more descriptive captions, yet they can produce generic outputs and are prone to hallucination. In this paper, we propose a new method to enhance vision-language datasets for remote sensing by integrating maps as external data sources, enabling the generation of detailed, context-rich captions. Additionally, we present methods to measure and mitigate hallucinations in LLM-generated text. We introduce fMoW-mm, a multimodal dataset incorporating satellite imagery, maps, metadata, and text annotations. We demonstrate its effectiveness for automatic target recognition in few-shot settings, achieving superior performance compared to other vision-language remote sensing datasets.
Paper Structure (15 sections, 1 equation, 4 figures, 2 tables)

This paper contains 15 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Datasets for Remote Sensing
  4. Hallucination Metrics and Mitigation Strategies
  5. Multimodal Dataset Curation
  6. Functional Map of the World (fMoW-rgb)
  7. OpenStreetMap (OSM) Tile Retrieval
  8. Caption Generation with GPT-4o
  9. Multimodal Functional Map of the World (fMoW-mm)
  10. Hallucination Metric
  11. Experiments
  12. Ablations
  13. Few-Shot Object Detection with CLIP
  14. Conclusion
  15. Acknowledgments

Figures (4)

  • Figure 1: Comparison of captioning methods: Rule-based captions are limited in detail. Unimodal LLM captions are fluid but often generic. Wide-area scenes covering diverse structures and objects require semantically rich descriptions. We leverage the semantic density of maps to generate comprehensive and detailed captions.
  • Figure 2: fMoW-mm data curation pipeline
  • Figure 3: A sample from the fMoW-mm dataset. The generated caption accurately incorporates information from the satellite image, map, and metadata.
  • Figure 4: Ablations. (a) Map Resolution: Higher resolution reduces hallucination rates and uncertainty in generated captions. (b) Map Types: Using landmarks-only gives the best balance, reducing hallucinations while limiting uncertainty. (c) Prompt Ensembling: Combining captions from multiple prompts did not significantly impact the metrics, however increasing from 3 to 5 prompts may result in repeated hallucinations that propagate into the final caption.