Atom-Level Optical Chemical Structure Recognition with Limited Supervision

TL;DR

This work tackles optical chemical structure recognition by introducing AtomLenz, a data-efficient, atom-level OCSR system that jointly predicts atoms, bonds, charges, and stereocenters from images and reconstructs a chemically valid molecular graph. It combines a four-channel object-detection backbone with a graph constructor, and employs weakly supervised training (ProbKT*) plus an edit-correction mechanism to adapt to new domains using SMILES supervision alone, enhanced by the ChemExpert ensemble. Thorough experiments show strong performance on hand-drawn images and out-of-domain data, with superior atom-level localization and favorable data-efficiency compared to SMILES-only baselines; a curated hand-drawn dataset and code release support reproducibility. The approach advances practical chemical data extraction from diverse depictions, enabling scalable digitization of literature and notes with atom-level interpretability and reliability.

Abstract

Identifying the chemical structure from a graphical representation, or image, of a molecule is a challenging pattern recognition task that would greatly benefit drug development. Yet, existing methods for chemical structure recognition do not typically generalize well, and show diminished effectiveness when confronted with domains where data is sparse, or costly to generate, such as hand-drawn molecule images. To address this limitation, we propose a new chemical structure recognition tool that delivers state-of-the-art performance and can adapt to new domains with a limited number of data samples and supervision. Unlike previous approaches, our method provides atom-level localization, and can therefore segment the image into the different atoms and bonds. Our model is the first model to perform OCSR with atom-level entity detection with only SMILES supervision. Through rigorous and extensive benchmarking, we demonstrate the preeminence of our chemical structure recognition approach in terms of data efficiency, accuracy, and atom-level entity prediction.

Figures (11)

• Figure 1: Problem setup. Our chemical structure recognition method takes an input image and predict atom-level entities predictions (atoms, bonds, charges, and stereocenters). This rich annotation can then be used to construct the molecular graph. Atom-level entity prediction models, like ours, predict both the atom-level annotations and molecular graph from the image. In contrast, molecular graph predictions models only predict the molecular graph and do not provide any localization in the original image.
• Figure 2: Weakly supervised training. The weakly supervised training data set consists of images of molecular depictions paired with SMILES. The input image is used by the object detection backbone to predict the atom-level entities while the SMILES is used by ProbKT* and the edit-correction scheme. In the first phase, ProbKT* will perform backpropagation to update the object detection backbone using probabilistic reasoning. In the second phase, both ProbKT* and the edit-correction mechanism will generate pseudo-labels for the atom-level entities, which are used to retrain the object detection backbone.
• Figure 3: Image samples from the dataset. Different example samples for the different datasets used in experiments. The hand-drawn and ChemPix datasets are used to assess the domain adaptation and out-of-domain performance. The atom localization dataset is used for testing object localization.
• Figure 4: Count accuracies per type over images if type is present in image for hand-drawn test set. We observe errors of 'AtomLenz+EditKT*' and 'DECIMER fine-tuned' tend to occur on different samples. Combining both approaches in ChemExpert improves performance.
• Figure 5: Different example samples for the different datasets used in experiments.