Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change

Murathan Kurfalı, Shorouq Zahra, Joakim Nivre, Gabriele Messori

TL;DR

Climate-Eval addresses the need for a comprehensive, standardized benchmark for NLP tasks in the climate-change domain. It combines 13 datasets into 25 tasks, including a newly created Guardian Climate News Corpus, and uses the LM Evaluation Harness to enable consistent evaluation across open-source LLMs. The study provides systematic zero-shot and few-shot assessments of models from 2B to 70B parameters, revealing that few-shot prompts help on some tasks but many remain challenging (e.g., SciDCC, Climate-Change NER) and that in-domain pretraining yields limited universal gains. It offers a reproducible evaluation setup, task prompts, and a framework to extend climate-domain evaluation, with implications for researchers and practitioners deploying LLMs in climate-related contexts.

Abstract

Climate-Eval is a comprehensive benchmark designed to evaluate natural language processing models across a broad range of tasks related to climate change. Climate-Eval aggregates existing datasets along with a newly developed news classification dataset, created specifically for this release. This results in a benchmark of 25 tasks based on 13 datasets, covering key aspects of climate discourse, including text classification, question answering, and information extraction. Our benchmark provides a standardized evaluation suite for systematically assessing the performance of large language models (LLMs) on these tasks. Additionally, we conduct an extensive evaluation of open-source LLMs (ranging from 2B to 70B parameters) in both zero-shot and few-shot settings, analyzing their strengths and limitations in the domain of climate change.

Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change

TL;DR

Climate-Eval addresses the need for a comprehensive, standardized benchmark for NLP tasks in the climate-change domain. It combines 13 datasets into 25 tasks, including a newly created Guardian Climate News Corpus, and uses the LM Evaluation Harness to enable consistent evaluation across open-source LLMs. The study provides systematic zero-shot and few-shot assessments of models from 2B to 70B parameters, revealing that few-shot prompts help on some tasks but many remain challenging (e.g., SciDCC, Climate-Change NER) and that in-domain pretraining yields limited universal gains. It offers a reproducible evaluation setup, task prompts, and a framework to extend climate-domain evaluation, with implications for researchers and practitioners deploying LLMs in climate-related contexts.

Abstract

Climate-Eval is a comprehensive benchmark designed to evaluate natural language processing models across a broad range of tasks related to climate change. Climate-Eval aggregates existing datasets along with a newly developed news classification dataset, created specifically for this release. This results in a benchmark of 25 tasks based on 13 datasets, covering key aspects of climate discourse, including text classification, question answering, and information extraction. Our benchmark provides a standardized evaluation suite for systematically assessing the performance of large language models (LLMs) on these tasks. Additionally, we conduct an extensive evaluation of open-source LLMs (ranging from 2B to 70B parameters) in both zero-shot and few-shot settings, analyzing their strengths and limitations in the domain of climate change.

Paper Structure

This paper contains 13 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Climate-Eval Benchmark
  4. Datasets and Tasks
  5. Evaluation Setup and Metrics
  6. Evaluation of Open-Source LLMs
  7. Zero-shot vs. Few-shot Performance
  8. Analysis of tasks
  9. Impact of In-domain Training
  10. Conclusion
  11. Models
  12. CO2 Emission Related to Experiments
  13. Label distribution

Figures (3)

  • Figure 1: The distribution of articles by year in the Guardian Climate News Corpus, broken down by category.
  • Figure 2: Average model performance across all tasks in Zero-Shot and 5-Shot settings.
  • Figure 3: The normalized distribution of labels in each task across all sets (train, development, and test).