UniMAP: Universal SMILES-Graph Representation Learning

TL;DR

Experimental results show that UniMAP outperforms current state-of-the-art pre-training methods on various downstream tasks, i.e. molecular property prediction, drug-target affinity prediction and drug-drug interaction.

Abstract

Molecular representation learning is fundamental for many drug related applications. Most existing molecular pre-training models are limited in using single molecular modality, either SMILES or graph representation. To effectively leverage both modalities, we argue that it is critical to capture the fine-grained 'semantics' between SMILES and graph, because subtle sequence/graph differences may lead to contrary molecular properties. In this paper, we propose a universal SMILE-graph representation learning model, namely UniMAP. Firstly, an embedding layer is employed to obtain the token and node/edge representation in SMILES and graph, respectively. A multi-layer Transformer is then utilized to conduct deep cross-modality fusion. Specially, four kinds of pre-training tasks are designed for UniMAP, including Multi-Level Cross-Modality Masking (CMM), SMILES-Graph Matching (SGM), Fragment-Level Alignment (FLA), and Domain Knowledge Learning (DKL). In this way, both global (i.e. SGM and DKL) and local (i.e. CMM and FLA) alignments are integrated to achieve comprehensive cross-modality fusion. We evaluate UniMAP on various downstream tasks, i.e. molecular property prediction, drug-target affinity prediction and drug-drug interaction. Experimental results show that UniMAP outperforms current state-of-the-art pre-training methods.We also visualize the learned representations to demonstrate the effect of multi-modality integration.

Figures (7)

• Figure 1: Two examples to demonstrate that substitution of a single fragment will lead to opposite properties in HIV dataset from MoleculeNet wu2018moleculenet. Specifically, the green parts in graph and the underlined parts in SMILES indicate the differences between the two molecules. A molecule is labeled as HIV active if it can inhibit HIV replication and HIV inactive otherwise.
• Figure 2: Overview of UniMAP. SMILES and Graph are processed by a shared Transformer to get a unified representation, which is supervised by both fragment-level and molecular-level pre-training tasks.
• Figure 3: Visualization of molecular embeddings with colors indicating different scaffolds.
• Figure 4: Visualization of fragment embeddings. Several clusters are labeled with colors indicating different categories, of which chemical names following several representative fragments are listed on the right.
• Figure 5: Visualization of attentions from SMILES to graph.