Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing LLMs for Governance with Human Oversight: Evaluating and Aligning LLMs on Expert Classification of Climate Misinformation for Detecting False or Misleading Claims about Climate Change

Mowafak Allaham, Ayse D. Lokmanoglu, P. Sol Hart, Erik C. Nisbet

TL;DR

This work investigates how LLMs can support governance of climate misinformation under human oversight. It benchmarks open-source and proprietary models on the CARDS expert-annotated dataset, compares them to climate-focused tools, and demonstrates that fine-tuning GPT-3.5-turbo on CARDS can match expert classifications in social-media content, outperforming several baselines. The study highlights the critical role of expert-annotated data and human oversight for domain-specific claim classification and shows potential for extending these approaches to governance tasks beyond climate. Overall, the findings suggest a practical path where high-performing LLMs, guided by expert-augmented datasets, assist civil-society and policy organizations in scalable misinformation detection and classification.

Abstract

Climate misinformation is a problem that has the potential to be substantially aggravated by the development of Large Language Models (LLMs). In this study we evaluate the potential for LLMs to be part of the solution for mitigating online dis/misinformation rather than the problem. Employing a public expert annotated dataset and a curated sample of social media content we evaluate the performance of proprietary vs. open source LLMs on climate misinformation classification task, comparing them to existing climate-focused computer-assisted tools and expert assessments. Results show (1) open-source models substantially under-perform in classifying climate misinformation compared to proprietary models, (2) existing climate-focused computer-assisted tools leveraging expert-annotated datasets continues to outperform many of proprietary models, including GPT-4o, and (3) demonstrate the efficacy and generalizability of fine-tuning GPT-3.5-turbo on expert annotated dataset in classifying claims about climate change at the equivalency of climate change experts with over 20 years of experience in climate communication. These findings highlight 1) the importance of incorporating human-oversight, such as incorporating expert-annotated datasets in training LLMs, for governance tasks that require subject-matter expertise like classifying climate misinformation, and 2) the potential for LLMs in facilitating civil society organizations to engage in various governance tasks such as classifying false or misleading claims in domains beyond climate change such as politics and health science.

Enhancing LLMs for Governance with Human Oversight: Evaluating and Aligning LLMs on Expert Classification of Climate Misinformation for Detecting False or Misleading Claims about Climate Change

TL;DR

This work investigates how LLMs can support governance of climate misinformation under human oversight. It benchmarks open-source and proprietary models on the CARDS expert-annotated dataset, compares them to climate-focused tools, and demonstrates that fine-tuning GPT-3.5-turbo on CARDS can match expert classifications in social-media content, outperforming several baselines. The study highlights the critical role of expert-annotated data and human oversight for domain-specific claim classification and shows potential for extending these approaches to governance tasks beyond climate. Overall, the findings suggest a practical path where high-performing LLMs, guided by expert-augmented datasets, assist civil-society and policy organizations in scalable misinformation detection and classification.

Abstract

Climate misinformation is a problem that has the potential to be substantially aggravated by the development of Large Language Models (LLMs). In this study we evaluate the potential for LLMs to be part of the solution for mitigating online dis/misinformation rather than the problem. Employing a public expert annotated dataset and a curated sample of social media content we evaluate the performance of proprietary vs. open source LLMs on climate misinformation classification task, comparing them to existing climate-focused computer-assisted tools and expert assessments. Results show (1) open-source models substantially under-perform in classifying climate misinformation compared to proprietary models, (2) existing climate-focused computer-assisted tools leveraging expert-annotated datasets continues to outperform many of proprietary models, including GPT-4o, and (3) demonstrate the efficacy and generalizability of fine-tuning GPT-3.5-turbo on expert annotated dataset in classifying claims about climate change at the equivalency of climate change experts with over 20 years of experience in climate communication. These findings highlight 1) the importance of incorporating human-oversight, such as incorporating expert-annotated datasets in training LLMs, for governance tasks that require subject-matter expertise like classifying climate misinformation, and 2) the potential for LLMs in facilitating civil society organizations to engage in various governance tasks such as classifying false or misleading claims in domains beyond climate change such as politics and health science.
Paper Structure (28 sections, 2 figures, 4 tables)

This paper contains 28 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data
  4. Evaluating LLMs using CARDS
  5. Social media dataset
  6. Methodology
  7. Benchmarking LLMs using CARDS
  8. Fine-tuning GPT-3.5-turbo on CARDS
  9. Classifying claims in social media data
  10. Expert annotation of claims in social media data
  11. Results
  12. Evaluating the performance of LLMs on CARDS data
  13. LLM vs. Expert classification of claims
  14. Conclusion
  15. Appendix
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Taxonomy of false or misleading claims published by coan2021computer
  • Figure 2: Proportion of valid responses for each LLM in the zero-shot task described in section \ref{['4.1']} to classify false or misleading claims about climate change in the CARDS test dataset. Numbers inside the bar plot represent the number of invalid responses by each model.