ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models

TL;DR

This paper introduces ClimaGen, an adaptive framework for generating climate science benchmarks with domain-expert input, and ClimaQA, a climate QA benchmark with gold and silver datasets designed to evaluate LLMs on MCQ, Freeform, and Cloze tasks. It defines three complexity levels and tailored metrics, including a factual-accuracy entailment score and phrase similarity for Cloze answers. Experiments across multiple models and settings show retrieval-augmented generation and task-specific fine-tuning improve performance, with GPT-4o often leading and source-based RAG providing the strongest gains. The work provides public datasets and code, enabling reproducible evaluation and scalable grounding of climate knowledge in LLMs.

Abstract

The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. The source code is publicly available at https://github.com/Rose-STL-Lab/genie-climaqa

Figures (8)

• Figure 1: ClimaGen - Our proposed Automated Benchmark Creation Framework. The QA generation framework creates synthetic data from seed contexts extracted from graduate-level textbooks using LLMs to generate base-level question-answer pairs and evolve them by adding complexities to the same. These are validated by domain experts during the annotation process to produce the semi-synthetic benchmark. The evaluator model is trained actively using the human-labeled examples in order to completely automate the process.
• Figure 2: Examples of Question Evolution. The first is the initial version of the generated question. The second is the enhanced version of the question that requires scientific reasoning to answer. The third is the modified version of the question that involves a hypothetical scenario. The contexts from the textbook data were used during the question evolution.
• Figure 3: Examples of the three types of scientific question-answering tasks presented in our benchmark
• Figure 4: Mean Phrase Similarity for Correctly Answered and Incorrectly Answered Cloze Questions
• Figure 5: Analysis of various LLMs under default setting on different tasks and different complexities. The first figure shows accuracy of models in the MCQ task while the others show different metrics under the Freeform task