Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule

Yi Xiao, Xiangxin Zhou, Qiang Liu, Liang Wang

TL;DR

This paper presents the first systematic survey on multimodal frameworks for molecules research, beginning with the development of molecular deep learning and pointing out the necessity to involve textual modality.

Abstract

Artificial intelligence has demonstrated immense potential in scientific research. Within molecular science, it is revolutionizing the traditional computer-aided paradigm, ushering in a new era of deep learning. With recent progress in multimodal learning and natural language processing, an emerging trend has targeted at building multimodal frameworks to jointly model molecules with textual domain knowledge. In this paper, we present the first systematic survey on multimodal frameworks for molecules research. Specifically,we begin with the development of molecular deep learning and point out the necessity to involve textual modality. Next, we focus on recent advances in text-molecule alignment methods, categorizing current models into two groups based on their architectures and listing relevant pre-training tasks. Furthermore, we delves into the utilization of large language models and prompting techniques for molecular tasks and present significant applications in drug discovery. Finally, we discuss the limitations in this field and highlight several promising directions for future research.

Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule

TL;DR

This paper presents the first systematic survey on multimodal frameworks for molecules research, beginning with the development of molecular deep learning and pointing out the necessity to involve textual modality.

Abstract

Artificial intelligence has demonstrated immense potential in scientific research. Within molecular science, it is revolutionizing the traditional computer-aided paradigm, ushering in a new era of deep learning. With recent progress in multimodal learning and natural language processing, an emerging trend has targeted at building multimodal frameworks to jointly model molecules with textual domain knowledge. In this paper, we present the first systematic survey on multimodal frameworks for molecules research. Specifically,we begin with the development of molecular deep learning and point out the necessity to involve textual modality. Next, we focus on recent advances in text-molecule alignment methods, categorizing current models into two groups based on their architectures and listing relevant pre-training tasks. Furthermore, we delves into the utilization of large language models and prompting techniques for molecular tasks and present significant applications in drug discovery. Finally, we discuss the limitations in this field and highlight several promising directions for future research.
Paper Structure (41 sections, 5 equations, 1 figure, 1 table)

This paper contains 41 sections, 5 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Molecular Descriptors and Encoding
  3. Small Molecule Representation
  4. 1D Sequences
  5. 2D Graph
  6. 3D Geometry
  7. Protein Representation
  8. Protein Sequence
  9. Protein Graph
  10. Latent Space Alignment between Text and Molecule
  11. Model Architecture
  12. Single-Stream Architecture
  13. Multi-Stream Architecture
  14. Pre-training Tasks
  15. Molecule-Text Contrastive Learning
  16. ...and 26 more sections

Figures (1)

  • Figure 1: Pipeline of multimodal framework for molecule and downstream molecular tasks (a-c). (a) Latent space alignment and adaptation of downstream tasks. The single-stream framework jointly models text and molecules with the same encoder. The downstream tasks are realized with task-specific prompts described in section \ref{['prompt']}; The multi-stream framework involves cross-modal alignment between text and molecules. Features from latent space can be directly used for tasks or be used in instruction-tuning. (b) Building a semi-autonomous agent for molecular research with instructions and in-context examples. (c) Building autonomous agent for chemistry with instructions and chain-of-thought prompting. Equipping agent with external tools and memory largely expand the autonomous level and capabilities.