EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models

TL;DR

EnviroExam introduces a domain-specific benchmark for environmental science LLMs by grounding evaluation in university curricula across 42 core courses and 936 questions. It uses a curriculum-based, 0-shot and 5-shot testing regime over 31 open-source LLMs, with a scoring pipeline that computes the mean, variability, and a composite index to compare models. The study reveals that a majority of models perform better with few-shot prompts but that chain-of-thought prompting yields mixed results, highlighting both promising gains and limitations for domain-specific AI in environmental science. The framework provides practical guidance for model selection and fine-tuning, and outlines future work to expand the dataset with domain textbooks while mitigating data-leakage concerns.

Abstract

In the field of environmental science, it is crucial to have robust evaluation metrics for large language models to ensure their efficacy and accuracy. We propose EnviroExam, a comprehensive evaluation method designed to assess the knowledge of large language models in the field of environmental science. EnviroExam is based on the curricula of top international universities, covering undergraduate, master's, and doctoral courses, and includes 936 questions across 42 core courses. By conducting 0-shot and 5-shot tests on 31 open-source large language models, EnviroExam reveals the performance differences among these models in the domain of environmental science and provides detailed evaluation standards. The results show that 61.3% of the models passed the 5-shot tests, while 48.39% passed the 0-shot tests. By introducing the coefficient of variation as an indicator, we evaluate the performance of mainstream open-source large language models in environmental science from multiple perspectives, providing effective criteria for selecting and fine-tuning language models in this field. Future research will involve constructing more domain-specific test sets using specialized environmental science textbooks to further enhance the accuracy and specificity of the evaluation.

Figures (9)

• Figure 1: Overview diagram of EnviroExam. Treat different large language models as different students, assess them and rank them accordingly.
• Figure 2: Data collection and process workflows
• Figure 3: The composite index scores $I$ of 31 LLMs evaluated on EnviroExam(5-shot).
• Figure 4: The composite index scores $I$ of 31 LLMs evaluated on EnviroExam(0-shot).
• Figure 5: A radar chart of the composite index scores $I$ for 31 large language models evaluated in 5-shot and 0-shot settings on enviroexam.