ChemPro: A Progressive Chemistry Benchmark for Large Language Models

TL;DR

ChemPro addresses the need for robust evaluation of foundational chemistry reasoning in LLMs by introducing a curriculum-aligned benchmark spanning four chemistry subfields and four difficulty levels, totaling 4,100 questions. It enforces provenance via established curricula (NCERT and JEE) and uses a dual MCQ and numerical format to assess conceptual and computational skills, organized with a progression across levels. Across 45 models, results reveal systematic degradation in accuracy as articulation and multi-step reasoning requirements increase, indicating that scale alone cannot reach human-level reliability, especially for numericals. The work identifies subfield bottlenecks, reveals limited efficacy of current agentic methods, and provides a framework to guide the next generation of chemistry-capable LLMs with real educational value.

Abstract

We introduce ChemPro, a progressive benchmark with 4100 natural language question-answer pairs in Chemistry, across 4 coherent sections of difficulty designed to assess the proficiency of Large Language Models (LLMs) in a broad spectrum of general chemistry topics. We include Multiple Choice Questions and Numerical Questions spread across fine-grained information recall, long-horizon reasoning, multi-concept questions, problem-solving with nuanced articulation, and straightforward questions in a balanced ratio, effectively covering Bio-Chemistry, Inorganic-Chemistry, Organic-Chemistry and Physical-Chemistry. ChemPro is carefully designed analogous to a student's academic evaluation for basic to high-school chemistry. A gradual increase in the question difficulty rigorously tests the ability of LLMs to progress from solving basic problems to solving more sophisticated challenges. We evaluate 45+7 state-of-the-art LLMs, spanning both open-source and proprietary variants, and our analysis reveals that while LLMs perform well on basic chemistry questions, their accuracy declines with different types and levels of complexity. These findings highlight the critical limitations of LLMs in general scientific reasoning and understanding and point towards understudied dimensions of difficulty, emphasizing the need for more robust methodologies to improve LLMs.

Figures (19)

• Figure 1: Overview of ChemPro.Left: A comparison with existing benchmarks; the y-axis represents observed difficulty for LLMs, the x-axis represents the traditional academic succession (elementary to expert), and bubble size is proportional to the number of questions in the dataset. E, M, C, and D correspond to Easy, Medium, Challenging, and Difficult sections of ChemPro. Center: Performance (Accuracy, y-axis) of all 40 open-source models evaluated on ChemPro MCQs showing the impact of model-size (x-axis) on performance (lines are Performance on individual ChemPro sections and columns is the overall average performance). Right: Exact-match accuracy (x-axis) vs Tolerance-based accuracy (y-axis) on ChemPro Numerical for all 40 open-source models (bubble size represents model parameter count).
• Figure 2: ChemPro benchmark structure: The benchmark spans four chemistry subfields (Biochemistry, Inorganic, Organic, Physical Chemistry) across four sections of difficulty ($\mathcal{CP}_E$, $\mathcal{CP}_M$, $\mathcal{CP}_C$, $\mathcal{CP}_D$) with balanced distribution of MCQs and numerical problems. Complete category distribution details are provided in Appendix.
• Figure 3: ChemPro Benchmark Curation Process. Visual workflow showing the systematic approach for creating ChemPro benchmark, from source collection across different difficulty tiers to quality validation and final dataset compilation. The process ensures source-aligned difficulty provenance while maintaining rigorous quality standards through multiple validation layers (both AI and Human).
• Figure 4: Parameter Scaling Limitations. Scaling benefits plateau well below human performance levels (90%+ expected accuracy). Numerical reasoning limitations persist across all parameter scales (x-axis: Parameters in billions). Left: Average Model Performance on Numericals (y-axis: Exact Match score). Center: Average Model Performance on Numericals (y-axis: Tolerance-Based score). Right: Average Model Performance on MCQs (y-axis: Accuracy).
• Figure 5: Comprehensive Performance Analysis.Left: Overall tolerance-based performance across all model sizes showing scaling limitations. Center: OpenAI model performance on numerical problems across ChemPro sections (y-axis: Tolerance-Based score). Right: OpenAI model performance on multiple-choice questions across ChemPro sections (y-axis: Accuracy).