GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition

TL;DR

OCSR requires reliable conversion of molecular images to machine-readable formats, including complex and hand-drawn depictions. The paper introduces GTR-VL, a graph-parsing VLM that uses Graph Traversal as Visual CoT and Faithfully Recognize What You've Seen, paired with GRPO reinforcement learning to handle hand-drawn data. It builds the GTR-1.3M dataset and MolRec-Bench, and adopts a two-stage training regime (SFT for printed, GRPO for hand-drawn) that yields state-of-the-art results across printed and hand-drawn benchmarks. These innovations improve interpretability, handling of abbreviated forms, and robustness to real-world chemical diagrams, including Markush structures, with practical impact on cheminformatics workflows and data digitization.

Abstract

Optical Chemical Structure Recognition (OCSR) is essential for converting molecular images into machine-readable formats. While recent vision-language models (VLMs) have shown promise, their image-captioning approach often struggles with complex molecular structures and inconsistent annotations. To address these issues, we introduce GTR-VL, featuring two key innovations: (1) the \textit{Graph Traversal as Visual Chain of Thought} mechanism that emulates human reasoning by incrementally parsing molecular graphs through sequential atom-bond predictions, and (2) the data-centric \textit{Faithfully Recognize What You've Seen} principle, which aligns abbreviated structures in images with their expanded annotations. For hand-drawn OCSR tasks, where datasets lack graph annotations and only provide final SMILES, we apply reinforcement learning using the GRPO method, introducing reward mechanisms like format reward, graph reward, and SMILES reward. This approach significantly enhances performance in hand-drawn recognition tasks through weak supervision. We developed GTR-1.3M, a large-scale instruction-tuning dataset with corrected annotations, and MolRec-Bench, the first benchmark for fine-grained evaluation of graph-parsing accuracy in OCSR. Our two-stage training scheme involves SFT training for printed images and the GRPO method for transferring capabilities to hand-drawn tasks. Experiments show that GTR-VL outperforms specialist models, chemistry-domain VLMs, and commercial VLMs on both printed and hand-drawn datasets.

Figures (19)

• Figure 1: The comparison of three paradigms. (a) Image-captioning approach: Directly generates SMILES from molecular structure images using either VLMs or specialist models. (b) Graph-parsing approach: Predict atoms and bonds in separate stages to construct a molecular graph, which is then converted to SMILES. (c) Ours: Jointly generates atoms and bonds to form a molecular graph, followed by SMILES generation. The graph is then used to construct a graph-based SMILES.
• Figure 2: Illustration of limitations in SMILES-based evaluation. (a) The positive example: both the predicted graph and SMILES match with the ground truth. (b) The counterexample: the graph-parsing OCSR algorithm correctly interprets the molecular graph, but the SMILES does not match the ground truth and is incorrectly judged as a prediction error.
• Figure 3: Our method is designed to achieve robust recognition across both printed and hand-drawn molecular structures. We begin by performing supervised fine-tuning (SFT) on the large-scale dataset GTR-1.3M to train a base model capable of recognizing printed molecular depictions. To further enhance the model’s ability to handle the challenging domain of hand-drawn inputs, we apply reinforcement learning using the GRPO algorithm on the DECIMER Hand-drawn dataset. This two-stage training pipeline enables the model to generalize effectively across different molecular representation styles.
• Figure 4: Comparison of model predictions on molecular images with abbreviated structures. Our model accurately retains abbreviations as superatoms (d) compared with (b) and (c).
• Figure 5: Comparison of model predictions on hand-drawn molecular images. Compared with DECIMER (b), our method (c) achieves more accurate predictions.