MolNexTR: A Generalized Deep Learning Model for Molecular Image Recognition

TL;DR

MolNexTR is a novel image-to-graph deep learning model that incorporates a unique dual-stream encoder to extract complex molecular image features, and combines chemical rules to predict atoms and bonds while understanding atom and bond layout rules.

Abstract

In the field of chemical structure recognition, the task of converting molecular images into machine-readable data formats such as SMILES string stands as a significant challenge, primarily due to the varied drawing styles and conventions prevalent in chemical literature. To bridge this gap, we proposed MolNexTR, a novel image-to-graph deep learning model that collaborates to fuse the strengths of ConvNext, a powerful Convolutional Neural Network variant, and Vision-TRansformer. This integration facilitates a more detailed extraction of both local and global features from molecular images. MolNexTR can predict atoms and bonds simultaneously and understand their layout rules. It also excels at flexibly integrating symbolic chemistry principles to discern chirality and decipher abbreviated structures. We further incorporate a series of advanced algorithms, including an improved data augmentation module, an image contamination module, and a post-processing module for getting the final SMILES output. These modules cooperate to enhance the model's robustness to diverse styles of molecular images found in real literature. In our test sets, MolNexTR has demonstrated superior performance, achieving an accuracy rate of 81-97%, marking a significant advancement in the domain of molecular structure recognition.

Figures (9)

• Figure 1: The same molecule represented by three different styles. (a) depicts the full molecular structure and does not contain chirality information. (b) and (c) specify the chirality and use different abbreviations and colors.
• Figure 2: The illustration of the molecular augmentation. The Molecular Augmentation consists of four main actions: 1) replace functional group, 2) add complex abbreviations, 3) add C bond, 4) add R-group.
• Figure 3: The illustration of the image contamination algorithm. The Contamination Algorithm consists of six main actions which are 1) add atom noise, 2) add bond noise, 3) add incomplete structural noise, 4) add line noise, 5) add incomplete atom noise, and 6) add arrow noise.
• Figure 4: Overview of our MolNexTR model. The molecular image is first encoded with the dual-stream encoder. Then a transformer-based decoder is applied for sequential atom and bond prediction. Finally, the post-processing modules ensure that the various molecular structures can be accurately converted to SMILES, SMART or MOLfile formats.
• Figure 5: Comparison of our model’s results with MolScribe on molecules with chirality or abbreviations.