Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SUPERChem: A Multimodal Reasoning Benchmark in Chemistry

Zehua Zhao, Zhixian Huang, Junren Li, Siyu Lin, Junting Zhou, Fengqi Cao, Kun Zhou, Rui Ge, Tingting Long, Yuexiang Zhu, Yan Liu, Jie Zheng, Junnian Wei, Rong Zhu, Peng Zou, Wenyu Li, Zekai Cheng, Tian Ding, Yaxuan Wang, Yizhao Yan, Tingru Wei, Haowei Ming, Weijie Mao, Chen Sun, Yiming Liu, Zichen Wang, Zuo Zhang, Tong Yang, Hao Ma, Zhen Gao, Jian Pei

TL;DR

SUPERChem tackles deficiencies in chemistry reasoning benchmarks by delivering a 500-question, expert-curated, multimodal benchmark with text-only equivalents and a formal reasoning-path evaluation via Reasoning Path Fidelity (RPF). The study demonstrates clear model-tier stratification, reveals substantial latent knowledge through pass@k analyses, and shows modality effects that are model-dependent. A comprehensive cognitive-diagnosis framework dissects where frontier models struggle, especially in synthesis and mechanistic understanding, through breakpoint and granular analyses. Together, SUPERChem provides a high-resolution platform for advancing AI chemistry toward expert-level reasoning and sets the stage for dynamic, future-oriented evaluation, including agent-enabled tasks and living benchmark updates.

Abstract

Current benchmarks for evaluating the chemical reasoning capabilities of Large Language Models (LLMs) are limited by oversimplified tasks, lack of process-level evaluation, and misalignment with expert-level chemistry skills. To address these issues, we introduce SUPERChem, a benchmark of 500 expert-curated reasoning-intensive chemistry problems, covering diverse subfields and provided in both multimodal and text-only formats. Original content and an iterative curation pipeline eliminate flawed items and mitigate data contamination. Each problem is paired with an expert-authored solution path, enabling Reasoning Path Fidelity (RPF) scoring to evaluate reasoning quality beyond final-answer accuracy. Evaluations against a human baseline of 40.3% accuracy show that even the best-performing model, GPT-5 (High), reaches only 38.5%, followed closely by Gemini 2.5 Pro (37.9%) and DeepSeek-V3.1-Think (37.3%). SUPERChem elicits multi-step, multimodal reasoning, reveals model-dependent effects of visual information, and distinguishes high-fidelity reasoners from heuristic ones. By providing a challenging benchmark and a reliable evaluation framework, SUPERChem aims to facilitate the advancement of LLMs toward expert-level chemical intelligence. The dataset of the benchmark is available at https://huggingface.co/datasets/ZehuaZhao/SUPERChem.

SUPERChem: A Multimodal Reasoning Benchmark in Chemistry

TL;DR

SUPERChem tackles deficiencies in chemistry reasoning benchmarks by delivering a 500-question, expert-curated, multimodal benchmark with text-only equivalents and a formal reasoning-path evaluation via Reasoning Path Fidelity (RPF). The study demonstrates clear model-tier stratification, reveals substantial latent knowledge through pass@k analyses, and shows modality effects that are model-dependent. A comprehensive cognitive-diagnosis framework dissects where frontier models struggle, especially in synthesis and mechanistic understanding, through breakpoint and granular analyses. Together, SUPERChem provides a high-resolution platform for advancing AI chemistry toward expert-level reasoning and sets the stage for dynamic, future-oriented evaluation, including agent-enabled tasks and living benchmark updates.

Abstract

Current benchmarks for evaluating the chemical reasoning capabilities of Large Language Models (LLMs) are limited by oversimplified tasks, lack of process-level evaluation, and misalignment with expert-level chemistry skills. To address these issues, we introduce SUPERChem, a benchmark of 500 expert-curated reasoning-intensive chemistry problems, covering diverse subfields and provided in both multimodal and text-only formats. Original content and an iterative curation pipeline eliminate flawed items and mitigate data contamination. Each problem is paired with an expert-authored solution path, enabling Reasoning Path Fidelity (RPF) scoring to evaluate reasoning quality beyond final-answer accuracy. Evaluations against a human baseline of 40.3% accuracy show that even the best-performing model, GPT-5 (High), reaches only 38.5%, followed closely by Gemini 2.5 Pro (37.9%) and DeepSeek-V3.1-Think (37.3%). SUPERChem elicits multi-step, multimodal reasoning, reveals model-dependent effects of visual information, and distinguishes high-fidelity reasoners from heuristic ones. By providing a challenging benchmark and a reliable evaluation framework, SUPERChem aims to facilitate the advancement of LLMs toward expert-level chemical intelligence. The dataset of the benchmark is available at https://huggingface.co/datasets/ZehuaZhao/SUPERChem.

Paper Structure

This paper contains 35 sections, 42 equations, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General Multi-disciplinary Reasoning Benchmarks
  4. Chemistry-specific Benchmarks
  5. Method
  6. Problem Formulation
  7. Pipeline
  8. Results and Analysis
  9. Benchmark Curation for a Rigorous Evaluation
  10. Overall Performance Hierarchy and Model Stratification
  11. Exploring the Latent Knowledge with Multiple Attempts
  12. The Nuanced Impact of Multimodality in Chemical Reasoning
  13. Deconstructing the Reasoning Process with Reasoning Path Fidelity
  14. Cognitive Diagnosis: From Benchmark Demands to Model Bottlenecks
  15. The Cognitive Landscape of SUPERChem
  16. ...and 20 more sections

Figures (11)

  • Figure 1: Overview of SUPERChem.
  • Figure 2: An example question of SUPERChem.
  • Figure 3: Data curation pipeline of SUPERChem. The pipeline involves three main stages: creation, review, and approval. Experts collaborate in a human-in-the-loop workflow to ensure the quality, correctness, and reasoning depth of the chemistry problems.
  • Figure 4: Frontier Model Performance with Multiple Attempts. The curves illustrate the pass@k accuracy as the number of attempts ($k$) increases from 1 to 8, revealing the latent potential of each model.
  • Figure 5: Performance comparison on the Multimodal-Essential Subset (238 Release Questions). The chart highlights the model-dependent effect of visual input on chemical reasoning accuracy.
  • ...and 6 more figures