Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeepMoLM: Leveraging Visual and Geometric Structural Information for Molecule-Text Modeling

Jing Lan, Hexiao Ding, Hongzhao Chen, Yufeng Jiang, Nga-Chun Ng, Gwing Kei Yip, Gerald W. Y. Cheng, Yunlin Mao, Jing Cai, Liang-ting Lin, Jung Sun Yoo

TL;DR

DeepMoLM tackles the challenge of aligning molecular images with 3D geometry and stereochemistry for downstream text tasks. It fuses a dual-path visual encoder with discrete 3D fingerprints (E3FP) via a cross-attention fusion projector to ground descriptions in conformer-aware structure. The two-stage training (Stage-1 alignment and Stage-2 instruction tuning) yields robust performance on molecule captioning, description, and property prediction, outperforming generalist baselines and approaching specialist methods while ensuring valid outputs. This grounding improves reliability and interpretability of molecule–text tasks, with implications for drug discovery and chemical literature synthesis.

Abstract

AI models for drug discovery and chemical literature mining must interpret molecular images and generate outputs consistent with 3D geometry and stereochemistry. Most molecular language models rely on strings or graphs, while vision-language models often miss stereochemical details and struggle to map continuous 3D structures into discrete tokens. We propose DeepMoLM: Deep Molecular Language M odeling, a dual-view framework that grounds high-resolution molecular images in geometric invariants derived from molecular conformations. DeepMoLM preserves high-frequency evidence from 1024 $\times$ 1024 inputs, encodes conformer neighborhoods as discrete Extended 3-Dimensional Fingerprints, and fuses visual and geometric streams with cross-attention, enabling physically grounded generation without atom coordinates. DeepMoLM improves PubChem captioning with a 12.3% relative METEOR gain over the strongest generalist baseline while staying competitive with specialist methods. It produces valid numeric outputs for all property queries and attains MAE 13.64 g/mol on Molecular Weight and 37.89 on Complexity in the specialist setting. On ChEBI-20 description generation from images, it exceeds generalist baselines and matches state-of-the-art vision-language models. Code is available at https://github.com/1anj/DeepMoLM.

DeepMoLM: Leveraging Visual and Geometric Structural Information for Molecule-Text Modeling

TL;DR

DeepMoLM tackles the challenge of aligning molecular images with 3D geometry and stereochemistry for downstream text tasks. It fuses a dual-path visual encoder with discrete 3D fingerprints (E3FP) via a cross-attention fusion projector to ground descriptions in conformer-aware structure. The two-stage training (Stage-1 alignment and Stage-2 instruction tuning) yields robust performance on molecule captioning, description, and property prediction, outperforming generalist baselines and approaching specialist methods while ensuring valid outputs. This grounding improves reliability and interpretability of molecule–text tasks, with implications for drug discovery and chemical literature synthesis.

Abstract

AI models for drug discovery and chemical literature mining must interpret molecular images and generate outputs consistent with 3D geometry and stereochemistry. Most molecular language models rely on strings or graphs, while vision-language models often miss stereochemical details and struggle to map continuous 3D structures into discrete tokens. We propose DeepMoLM: Deep Molecular Language M odeling, a dual-view framework that grounds high-resolution molecular images in geometric invariants derived from molecular conformations. DeepMoLM preserves high-frequency evidence from 1024 1024 inputs, encodes conformer neighborhoods as discrete Extended 3-Dimensional Fingerprints, and fuses visual and geometric streams with cross-attention, enabling physically grounded generation without atom coordinates. DeepMoLM improves PubChem captioning with a 12.3% relative METEOR gain over the strongest generalist baseline while staying competitive with specialist methods. It produces valid numeric outputs for all property queries and attains MAE 13.64 g/mol on Molecular Weight and 37.89 on Complexity in the specialist setting. On ChEBI-20 description generation from images, it exceeds generalist baselines and matches state-of-the-art vision-language models. Code is available at https://github.com/1anj/DeepMoLM.
Paper Structure (33 sections, 23 equations, 3 figures, 5 tables, 1 algorithm)

This paper contains 33 sections, 23 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Molecular Representations
  4. Vision Language Models for Molecular Images
  5. Geometric Learning and Multimodal Fusion
  6. Methodology
  7. Molecular Structure Representation
  8. Structure-aware 1D Molecular Tokens
  9. 3D Geometric Tokens
  10. Token Embedding Fusion
  11. Molecular DeepEncoder
  12. Local Window Vision Transformer
  13. Convolutional Token Compressor
  14. Global Attention Transformer
  15. Local and Global Feature Fusion
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Comparison of prior approaches with DeepMoLM.(A) Previous studies represented molecules using low-resolution images processed through Optical Chemical Structure Recognition, or with 2D molecular graphs. (B) DeepMoLM integrates high-resolution molecular images with explicit 3D conformer-based structural descriptors. Through a cross-attention mechanism that links visual and geometric modalities within a Vision-Language Model, DeepMoLM supports end-to-end execution of complex tasks such as 3D molecule captioning and property prediction.
  • Figure 2: Architecture of DeepMoLM. The framework performs dual-view molecular understanding in three stages. (1) DeepEncoder extracts multi-scale features from a high-resolution molecular image using a SAM-Base local encoder, convolutional token compression, and a CLIP-Large global encoder. (2) Fusion Projector uses cross-attention to fuse 2D visual tokens with discrete 3D geometric fingerprints. (3) VLM Decoder generates captions and property text from the fused representation.
  • Figure 3: VLM Interface for Open-Text Molecular QA