MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs

TL;DR

MolecularIQ presents the first fully symbolically verifiable benchmark for evaluating molecular-structure reasoning in LLMs, grounding evaluation in exact graph-based ground truths derived from RDKit. It defines three task types—feature counting, indexing, and constrained generation—across six feature families and three complexity axes (SMILES variants, Bertz complexity, multitask load), plus a dynamic variant MolecularIQD. The authors integrate MolecularIQ with standard evaluation frameworks, provide an open leaderboard, and report broad model comparisons across 38 LLMs, revealing that structural understanding remains a bottleneck and is sensitive to representation choices and task composition. The work offers a principled, reproducible platform to diagnose and guide the development of models that reason faithfully over molecular structure, while outlining concrete limitations and directions for extending the benchmark to 3D and multi-molecule scenarios.

Abstract

A molecule's properties are fundamentally determined by its composition and structure encoded in its molecular graph. Thus, reasoning about molecular properties requires the ability to parse and understand the molecular graph. Large Language Models (LLMs) are increasingly applied to chemistry, tackling tasks such as molecular name conversion, captioning, text-guided generation, and property or reaction prediction. Most existing benchmarks emphasize general chemical knowledge, rely on literature or surrogate labels that risk leakage or bias, or reduce evaluation to multiple-choice questions. We introduce MolecularIQ, a molecular structure reasoning benchmark focused exclusively on symbolically verifiable tasks. MolecularIQ enables fine-grained evaluation of reasoning over molecular graphs and reveals capability patterns that localize model failures to specific tasks and molecular structures. This provides actionable insights into the strengths and limitations of current chemistry LLMs and guides the development of models that reason faithfully over molecular structure.

Figures (21)

• Figure 1: Overview of MolecularIQ benchmark for measuring chemical structure reasoning capabilities of LLMs.
• Figure 2: MolecularIQ composition. (left) Number of questions by task type, multitask load, and Bertz complexity. (right) The percentage of features types present for counting, indexing, and generation.
• Figure 3: Performance profile of top 10 models for counting tasks.(a–d) Heatmaps of average success ratio per model (%) by (a) Bertz complexity of the molecule, (b) functional-group families (ROH: hydroxyls, $\pi$-sys: $\pi$-systems, S: sulfurs, Hal: halides, Ar: aromatics, ROR: ethers, C=O/S: (thio)carbonyls, Alk: alkyls, NR: amines, C$\equiv$N/N=O: nitriles/nitros), (c) multitask load, and (d) molecular feature groups (C: composition, CP: chemical perception, CT: chemistry-typed topology, FG: functional groups, SYN: synthesis). (e–f) Success/failure transitions averaged across top models for matched questions (e) as additional sub-tasks are added to the same prompt, and (f) when moving from counting to indexing variants of the same underlying property, probing whether correct counts are grounded in explicit substructure identification. (g) Sensitivity to input representation, comparing accuracy between different SMILES representations.
• Figure 11: Example molecules from each Bertz complexity bin, labeled with their Bertz complexity.
• Figure 12: Bertz complexity distribution of molecules in MolecularIQ (top), ChemCOTBench (middle), and ChemIQ (bottom).