MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation

TL;DR

MolLangBench introduces a structure-grounded benchmark for language-prompted molecular recognition, editing, and generation across multiple representations (graphs, SMILES, images). It combines automated ground-truth extraction with expert-annotated prompts to ensure determinism and reliability. Baseline evaluations reveal notable gaps in even state-of-the-art models, especially for generation and precise localization, underscoring the need for improved molecule-language bridging. The benchmark aims to catalyze progress toward reliable AI systems for chemical tasks by highlighting concrete weaknesses and providing rigorous evaluation standards.

Abstract

Precise recognition, editing, and generation of molecules are essential prerequisites for both chemists and AI systems tackling various chemical tasks. We present MolLangBench, a comprehensive benchmark designed to evaluate fundamental molecule-language interface tasks: language-prompted molecular structure recognition, editing, and generation. To ensure high-quality, unambiguous, and deterministic outputs, we construct the recognition tasks using automated cheminformatics tools, and curate editing and generation tasks through rigorous expert annotation and validation. MolLangBench supports the evaluation of models that interface language with different molecular representations, including linear strings, molecular images, and molecular graphs. Evaluations of state-of-the-art models reveal significant limitations: the strongest model (GPT-5) achieves $86.2\%$ and $85.5\%$ accuracy on recognition and editing tasks, which are intuitively simple for humans, and performs even worse on the generation task, reaching only $43.0\%$ accuracy. These results highlight the shortcomings of current AI systems in handling even preliminary molecular recognition and manipulation tasks. We hope MolLangBench will catalyze further research toward more effective and reliable AI systems for chemical applications.

Figures (26)

• Figure 1: Illustration of two simplified molecular design scenarios encountered by chemists: (a) molecule optimization and (b) de novo molecule design. Inspired by these workflows, we benchmark three molecule-language interface tasks: molecular structure recognition, molecule editing, and molecule generation.
• Figure 2: Annotation pipeline for molecule editing and generation tasks. The illustrated example is a simplified case for clarity; real annotations are much more complex.
• Figure 3: Illustration of a molecular structure recognition task. The molecular structure is shown in the top-left, the question prompt appears on the right, and the ground-truth is shown in the bottom-left. In this example, the molecule is represented as a SMILES string. An example using a molecular image as input is provided in the Appendix \ref{['appdix:recog_examples']}.
• Figure 4: Illustration of a molecule editing example. The starting molecule is shown in the top-left, the editing instruction appears on the right, and the ground-truth resulting molecule is shown in the bottom-left. In this example, the molecule is represented as a SMILES string. A corresponding example using a molecular image as input is provided in the Appendix \ref{['appdix:edit_examples']}.
• Figure 5: Illustration of a molecule generation example. Only the molecular description is shown on the right; the full input prompt is provided in the Appendix \ref{['appdix:gen_examples']} due to space constraints. The ground-truth resulting molecule is shown on the left.