Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models

Yuqing Huang, Rongyang Zhang, Xuesong He, Xuyang Zhi, Hao Wang, Xin Li, Feiyang Xu, Deguang Liu, Huadong Liang, Yi Li, Jian Cui, Zimu Liu, Shijin Wang, Guoping Hu, Guiquan Liu, Qi Liu, Defu Lian, Enhong Chen

TL;DR

ChemEval presents the first open, multi-level benchmark tailored to chemistry for evaluating large language models. It combines four progressive levels, twelve capability dimensions, and 42 tasks designed with domain-expert input to assess knowledge, literature understanding, molecular reasoning, and scientific deduction. The study reveals that general LLMs excel at literature tasks while specialized chemistry models perform better on domain-specific challenges, highlighting complementary strengths and the need for balanced training. By providing comprehensive data, prompts, and metrics, ChemEval aims to drive advances in chemical informatics and LLM evaluation, with open-source accessibility to foster community collaboration.

Abstract

There is a growing interest in the role that LLMs play in chemistry which lead to an increased focus on the development of LLMs benchmarks tailored to chemical domains to assess the performance of LLMs across a spectrum of chemical tasks varying in type and complexity. However, existing benchmarks in this domain fail to adequately meet the specific requirements of chemical research professionals. To this end, we propose \textbf{\textit{ChemEval}}, which provides a comprehensive assessment of the capabilities of LLMs across a wide range of chemical domain tasks. Specifically, ChemEval identified 4 crucial progressive levels in chemistry, assessing 12 dimensions of LLMs across 42 distinct chemical tasks which are informed by open-source data and the data meticulously crafted by chemical experts, ensuring that the tasks have practical value and can effectively evaluate the capabilities of LLMs. In the experiment, we evaluate 12 mainstream LLMs on ChemEval under zero-shot and few-shot learning contexts, which included carefully selected demonstration examples and carefully designed prompts. The results show that while general LLMs like GPT-4 and Claude-3.5 excel in literature understanding and instruction following, they fall short in tasks demanding advanced chemical knowledge. Conversely, specialized LLMs exhibit enhanced chemical competencies, albeit with reduced literary comprehension. This suggests that LLMs have significant potential for enhancement when tackling sophisticated tasks in the field of chemistry. We believe our work will facilitate the exploration of their potential to drive progress in chemistry. Our benchmark and analysis will be available at {\color{blue} \url{https://github.com/USTC-StarTeam/ChemEval}}.

ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models

TL;DR

ChemEval presents the first open, multi-level benchmark tailored to chemistry for evaluating large language models. It combines four progressive levels, twelve capability dimensions, and 42 tasks designed with domain-expert input to assess knowledge, literature understanding, molecular reasoning, and scientific deduction. The study reveals that general LLMs excel at literature tasks while specialized chemistry models perform better on domain-specific challenges, highlighting complementary strengths and the need for balanced training. By providing comprehensive data, prompts, and metrics, ChemEval aims to drive advances in chemical informatics and LLM evaluation, with open-source accessibility to foster community collaboration.

Abstract

There is a growing interest in the role that LLMs play in chemistry which lead to an increased focus on the development of LLMs benchmarks tailored to chemical domains to assess the performance of LLMs across a spectrum of chemical tasks varying in type and complexity. However, existing benchmarks in this domain fail to adequately meet the specific requirements of chemical research professionals. To this end, we propose \textbf{\textit{ChemEval}}, which provides a comprehensive assessment of the capabilities of LLMs across a wide range of chemical domain tasks. Specifically, ChemEval identified 4 crucial progressive levels in chemistry, assessing 12 dimensions of LLMs across 42 distinct chemical tasks which are informed by open-source data and the data meticulously crafted by chemical experts, ensuring that the tasks have practical value and can effectively evaluate the capabilities of LLMs. In the experiment, we evaluate 12 mainstream LLMs on ChemEval under zero-shot and few-shot learning contexts, which included carefully selected demonstration examples and carefully designed prompts. The results show that while general LLMs like GPT-4 and Claude-3.5 excel in literature understanding and instruction following, they fall short in tasks demanding advanced chemical knowledge. Conversely, specialized LLMs exhibit enhanced chemical competencies, albeit with reduced literary comprehension. This suggests that LLMs have significant potential for enhancement when tackling sophisticated tasks in the field of chemistry. We believe our work will facilitate the exploration of their potential to drive progress in chemistry. Our benchmark and analysis will be available at {\color{blue} \url{https://github.com/USTC-StarTeam/ChemEval}}.
Paper Structure (98 sections, 1 figure, 8 tables)

This paper contains 98 sections, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models
  4. Large Language Models for Chemistry
  5. Large Language Model Evaluations
  6. ChemEval
  7. Advanced Knowledge Question Answering
  8. Objective Questions
  9. Subjective Questions
  10. Literature Understanding
  11. Information Extraction
  12. Inductive Generation
  13. Molecular Understanding
  14. Molecular Name Generation
  15. Molecular Name Translation
  16. ...and 83 more sections

Figures (1)

  • Figure 1: Data Collection steps of ChemEval. The process is divided into three main steps: a). Data Collection: Raw data is collected from academic websites via web crawling, and experts manually gather data from professional textbooks and experimental data. b). Data Filtering: The raw data undergoes deduplication and removal of irrelevant items to produce filtered data. c). Q&A Pair Construction: Experts manually construct Q&A pairs related to chemistry and create prompt instructions, resulting in four instruction test sets.