Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MolXPT: Wrapping Molecules with Text for Generative Pre-training

Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, Tie-Yan Liu

TL;DR

MolXPT presents a unified GPT-style model pre-trained on scientific text, SMILES, and wrapped text–SMILES sequences to capture cross-modal molecular knowledge. By replacing molecule names in text with SMILES via NER-based linking, the model learns joint representations that improve molecular property prediction and molecule–text translation while using fewer parameters than large-scale baselines. Prompt-based finetuning enables a single backbone to handle diverse tasks, including zero-shot molecular generation, demonstrating practical cross-domain applicability. The work highlights the value of integrating textual contexts with chemical representations for more efficient, versatile molecular modeling.

Abstract

Generative pre-trained Transformer (GPT) has demonstrates its great success in natural language processing and related techniques have been adapted into molecular modeling. Considering that text is the most important record for scientific discovery, in this paper, we propose MolXPT, a unified language model of text and molecules pre-trained on SMILES (a sequence representation of molecules) wrapped by text. Briefly, we detect the molecule names in each sequence and replace them to the corresponding SMILES. In this way, the SMILES could leverage the information from surrounding text, and vice versa. The above wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem are all fed into a language model for pre-training. Experimental results demonstrate that MolXPT outperforms strong baselines of molecular property prediction on MoleculeNet, performs comparably to the best model in text-molecule translation while using less than half of its parameters, and enables zero-shot molecular generation without finetuning.

MolXPT: Wrapping Molecules with Text for Generative Pre-training

TL;DR

MolXPT presents a unified GPT-style model pre-trained on scientific text, SMILES, and wrapped text–SMILES sequences to capture cross-modal molecular knowledge. By replacing molecule names in text with SMILES via NER-based linking, the model learns joint representations that improve molecular property prediction and molecule–text translation while using fewer parameters than large-scale baselines. Prompt-based finetuning enables a single backbone to handle diverse tasks, including zero-shot molecular generation, demonstrating practical cross-domain applicability. The work highlights the value of integrating textual contexts with chemical representations for more efficient, versatile molecular modeling.

Abstract

Generative pre-trained Transformer (GPT) has demonstrates its great success in natural language processing and related techniques have been adapted into molecular modeling. Considering that text is the most important record for scientific discovery, in this paper, we propose MolXPT, a unified language model of text and molecules pre-trained on SMILES (a sequence representation of molecules) wrapped by text. Briefly, we detect the molecule names in each sequence and replace them to the corresponding SMILES. In this way, the SMILES could leverage the information from surrounding text, and vice versa. The above wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem are all fed into a language model for pre-training. Experimental results demonstrate that MolXPT outperforms strong baselines of molecular property prediction on MoleculeNet, performs comparably to the best model in text-molecule translation while using less than half of its parameters, and enables zero-shot molecular generation without finetuning.
Paper Structure (16 sections, 4 equations, 2 figures, 4 tables)

This paper contains 16 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Our Method
  3. Pre-training corpus
  4. Model and training
  5. Experiments
  6. Results on MoleculeNet
  7. Results on text-molecule translation
  8. Conclusions and Future Work
  9. Datasets and Baselines of MoleculeNet
  10. Pre-training hyper-parameters
  11. Finetuning details of downstream tasks
  12. Prompts for finetuning MoleculeNet
  13. Details of finetuning MoleculeNet
  14. Details of finetuning text-molecule generation
  15. MoleculeNet finetuning strategy selection
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Framework of MolXPT. MolXPT is pretrained on text from PubMed, SMILES from PubChem and wrapped sequences between SMILES and text. The wrapped sequences are obtained by applying NER and entity linking to text and then replacing matched molecular mentions with SMILES. MolXPT can be finetuned for various text and molecular downstream tasks, like molecular property prediction and molecule-text translation.
  • Figure 2: Examples for zero-shot text-to-molecule generation. We randomly pick up three cases that MolXPT can successfully generate the reference molecules without finetuning.