Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Climate Disinformation Detection: Integrating Vision-Language Models with External Knowledge Sources

Marzieh Adeli Shamsabad, Hamed Ghodrati

TL;DR

This work tackles climate disinformation by extending vision-language models with up-to-date external knowledge sources to better verify image–claim pairs. Using GPT-4o as the core multimodal reasoner, the authors retrieve provenance, claim context, and expert assessments from multiple sources and apply two prompting strategies (CoT and CoD) to classify content as Accurate, Misleading, False, or Unverifiable. They introduce the CliME dataset adaptation and show that combining all external sources yields the strongest factuality performance, notably achieving around 86% accuracy in a two-class setting and approximately 70% in the four-class setting, with zero rejections in some configurations. The approach demonstrates the value of external evidence in multimodal climate misinformation detection and points to future work on data expansion and retrieval optimization to improve robustness and scalability in real-world deployments.

Abstract

Climate disinformation has become a major challenge in today digital world, especially with the rise of misleading images and videos shared widely on social media. These false claims are often convincing and difficult to detect, which can delay actions on climate change. While vision-language models (VLMs) have been used to identify visual disinformation, they rely only on the knowledge available at the time of training. This limits their ability to reason about recent events or updates. The main goal of this paper is to overcome that limitation by combining VLMs with external knowledge. By retrieving up-to-date information such as reverse image results, online fact-checks, and trusted expert content, the system can better assess whether an image and its claim are accurate, misleading, false, or unverifiable. This approach improves the model ability to handle real-world climate disinformation and supports efforts to protect public understanding of science in a rapidly changing information landscape.

Multimodal Climate Disinformation Detection: Integrating Vision-Language Models with External Knowledge Sources

TL;DR

This work tackles climate disinformation by extending vision-language models with up-to-date external knowledge sources to better verify image–claim pairs. Using GPT-4o as the core multimodal reasoner, the authors retrieve provenance, claim context, and expert assessments from multiple sources and apply two prompting strategies (CoT and CoD) to classify content as Accurate, Misleading, False, or Unverifiable. They introduce the CliME dataset adaptation and show that combining all external sources yields the strongest factuality performance, notably achieving around 86% accuracy in a two-class setting and approximately 70% in the four-class setting, with zero rejections in some configurations. The approach demonstrates the value of external evidence in multimodal climate misinformation detection and points to future work on data expansion and retrieval optimization to improve robustness and scalability in real-world deployments.

Abstract

Climate disinformation has become a major challenge in today digital world, especially with the rise of misleading images and videos shared widely on social media. These false claims are often convincing and difficult to detect, which can delay actions on climate change. While vision-language models (VLMs) have been used to identify visual disinformation, they rely only on the knowledge available at the time of training. This limits their ability to reason about recent events or updates. The main goal of this paper is to overcome that limitation by combining VLMs with external knowledge. By retrieving up-to-date information such as reverse image results, online fact-checks, and trusted expert content, the system can better assess whether an image and its claim are accurate, misleading, false, or unverifiable. This approach improves the model ability to handle real-world climate disinformation and supports efforts to protect public understanding of science in a rapidly changing information landscape.
Paper Structure (10 sections, 5 figures, 5 tables)

This paper contains 10 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Approach
  4. Dataset
  5. Methodology
  6. Experiments
  7. Evaluation Metrics
  8. Results
  9. Discussion
  10. Conclusion and futur work

Figures (5)

  • Figure 1: Dataset samples and distribution for the 4-class setting
  • Figure 2: Dataset samples and distribution for the 2-class setting
  • Figure 4: Conditional Inclusion Reasoning
  • Figure 5: Confusion Matrix in Combination Source
  • Figure 6: Confusion Matrix in Combination Source