Towards 3D Molecule-Text Interpretation in Language Models

TL;DR

3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder, and significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties.

Abstract

Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, to enhance 3D-MoLM's ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties. We release our codes and datasets at https://github.com/lsh0520/3D-MoLM.

Figures (7)

• Figure 1: Demonstration of 3D-MoLM. 3D-MoLM is a general-purpose molecular LM that can be applied for molecule-text retrieval, molecule captioning, and molecular QA tasks. Flame denotes tunable modules, while snowflake indicates frozen modules.
• Figure 2: Illustration of 3D-MoLM's architectures at different stages.
• Figure 3: Illustration of the model architectures (upper part) and the dataset usage (bottom part) for the three training stages. PubChem is used for the stage 1 (i.e., 3D molecule-text representation learning) and stage 2 (i.e., 3D molecule-text alignment via generative learning). 3D-MoIT is used for 3D molecule-centric instruction tuning. Texts in the same color indicate the same information source.
• Figure 4: Utilizing GPT-3.5 to enrich the descriptions in the pretraining subset of PubChem Dataset.
• Figure 5: The prompt template for textual description enrichment and quality evaluation.