Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Debunking of Climate Misinformation

Francisco Zanartu, Yulia Otmakhova, John Cook, Lea Frermann

TL;DR

The paper addresses the spread of climate misinformation and proposes Generative Debunking, an LLM-based framework that outputs four-layer truth-sandwich debunkings. It integrates contrarian claim classification (CARDS) and fallacy detection (FLICC) with three prompting strategies across GPT-4, Palm2, and Mixtral, including a four-layer modular architecture with external data and tools. Key contributions include a gold-standard CARDS-examples dataset, evaluation results, and a public data/code/demo release, while highlighting challenges in factuality, relevancy, and evaluation by non-experts. The work advances scalable, automated climate-misinformation correction and lays out concrete directions for improving factual grounding and assessment in future research.

Abstract

Misinformation about climate change causes numerous negative impacts, necessitating corrective responses. Psychological research has offered various strategies for reducing the influence of climate misinformation, such as the fact-myth-fallacy-fact-structure. However, practically implementing corrective interventions at scale represents a challenge. Automatic detection and correction of misinformation offers a solution to the misinformation problem. This study documents the development of large language models that accept as input a climate myth and produce a debunking that adheres to the fact-myth-fallacy-fact (``truth sandwich'') structure, by incorporating contrarian claim classification and fallacy detection into an LLM prompting framework. We combine open (Mixtral, Palm2) and proprietary (GPT-4) LLMs with prompting strategies of varying complexity. Experiments reveal promising performance of GPT-4 and Mixtral if combined with structured prompts. We identify specific challenges of debunking generation and human evaluation, and map out avenues for future work. We release a dataset of high-quality truth-sandwich debunkings, source code and a demo of the debunking system.

Generative Debunking of Climate Misinformation

TL;DR

The paper addresses the spread of climate misinformation and proposes Generative Debunking, an LLM-based framework that outputs four-layer truth-sandwich debunkings. It integrates contrarian claim classification (CARDS) and fallacy detection (FLICC) with three prompting strategies across GPT-4, Palm2, and Mixtral, including a four-layer modular architecture with external data and tools. Key contributions include a gold-standard CARDS-examples dataset, evaluation results, and a public data/code/demo release, while highlighting challenges in factuality, relevancy, and evaluation by non-experts. The work advances scalable, automated climate-misinformation correction and lays out concrete directions for improving factual grounding and assessment in future research.

Abstract

Misinformation about climate change causes numerous negative impacts, necessitating corrective responses. Psychological research has offered various strategies for reducing the influence of climate misinformation, such as the fact-myth-fallacy-fact-structure. However, practically implementing corrective interventions at scale represents a challenge. Automatic detection and correction of misinformation offers a solution to the misinformation problem. This study documents the development of large language models that accept as input a climate myth and produce a debunking that adheres to the fact-myth-fallacy-fact (``truth sandwich'') structure, by incorporating contrarian claim classification and fallacy detection into an LLM prompting framework. We combine open (Mixtral, Palm2) and proprietary (GPT-4) LLMs with prompting strategies of varying complexity. Experiments reveal promising performance of GPT-4 and Mixtral if combined with structured prompts. We identify specific challenges of debunking generation and human evaluation, and map out avenues for future work. We release a dataset of high-quality truth-sandwich debunkings, source code and a demo of the debunking system.
Paper Structure (25 sections, 2 figures, 12 tables)

This paper contains 25 sections, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Psychologically effective debunking
  4. Automatic debunking
  5. Data
  6. FLICC test set
  7. CLIMATE-FEVER dataset
  8. CARDS-examples dataset
  9. Generative Debunking
  10. Single prompt, no context (GPT4)
  11. Single prompt, with context (Palm2)
  12. Structured prompt, with context (Mixtral)
  13. Layer 1: FACT.
  14. Layer 2: MYTH.
  15. Layer 3: FALLACY.
  16. ...and 10 more sections

Figures (2)

  • Figure 1: An example input myth (top, dark gray) and fact-myth-fallacy-fact ("truth sandwich") debunking generated by our model (bottom).
  • Figure 2: Overview of our dynamic prompting approaches. Left: Single prompt with dynamic fallacy prediction (FLICC) and example retrieval (CARDS). Right: Structured prompt with additional ReAct component (Fact 1) and FEVER evidence retrieval (Fact 2). External resources are shown as diamonds, and shared components between the two approaches are highlighted in green.