Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations

TL;DR

ChemCoTBench addresses the need for structured chemical reasoning assessment by representing molecular transformations as modular operations and evaluating LLMs on Molecular Property Optimization and Reaction Prediction. It provides a data-rich ChemCoTDataset and a taxonomy of reasoning steps, enabling slow-thinking, verifiable problem solving in chemistry. Experiments show that although reasoning-enabled models outperform non-reasoning ones on challenging chemical tasks, open-source models still lag due to limited domain-specific data, and domain-specific CoT data augmentation substantially improves performance. The work offers a practical benchmark and dataset to advance AI-assisted chemical discovery.

Abstract

While large language models (LLMs) with Chain-of-Thought (CoT) reasoning excel in mathematics and coding, their potential for systematic reasoning in chemistry, a domain demanding rigorous structural analysis for real-world tasks like drug design and reaction engineering, remains untapped. Current benchmarks focus on simple knowledge retrieval, neglecting step-by-step reasoning required for complex tasks such as molecular optimization and reaction prediction. To address this, we introduce ChemCoTBench, a reasoning framework that bridges molecular structure understanding with arithmetic-inspired operations, including addition, deletion, and substitution, to formalize chemical problem-solving into transparent, step-by-step workflows. By treating molecular transformations as modular "chemical operations", the framework enables slow-thinking reasoning, mirroring the logic of mathematical proofs while grounding solutions in real-world chemical constraints. We evaluate models on two high-impact tasks: Molecular Property Optimization and Chemical Reaction Prediction. These tasks mirror real-world challenges while providing structured evaluability. By providing annotated datasets, a reasoning taxonomy, and baseline evaluations, ChemCoTBench bridges the gap between abstract reasoning methods and practical chemical discovery, establishing a foundation for advancing LLMs as tools for AI-driven scientific innovation.

Figures (10)

• Figure 1: Previous chemical benchmarks focus on factual recall with domain knowledge, while our ChemCoTBench focuses on the evaluation of step-wise reasoning for complex chemical problems by defining a set of modular chemical operations.
• Figure 2: (a). Distribution analysis for ChemCoTBench. (b). Samples from both molecular understanding and editing tasks achieved exceptionally high accuracy in chemical expert evaluations of chemical entities, including function group names, molecule names, chemical operation names, reaction information, etc. (c). Samples from molecule optimization and reaction prediction also show high accuracy (> 89%) in chemical expert evaluations.
• Figure 3: The dataset construction pipeline of ChemCoTBench contains four steps, including raw data collection, molecule filtering and sampling, chain-of-thoughts annotation, and chemical expert review & refinement. We also visualize the samples from the four main tasks and their corresponding modular chemical operations during the reasoning process.
• Figure 4: The top two rows compare the reasoning performance of the Qwen-2.5-Instruct series against its DeepSeek-R1-distilled versions. The bottom row illustrates performance improvements in Qwen-2.5-Instruct when enhanced with the CoT template and detailed CoT process.
• Figure 5: Error distribution analysis for ring counting and functional-group counting tasks.