ProtCLIP: Function-Informed Protein Multi-Modal Learning

TL;DR

ProtCLIP addresses the gap in protein multi-modal learning by building ProtAnno, a large-scale protein-biotext dataset with a property-driven sampling strategy, and a function-informed pre-training paradigm that jointly optimizes global alignment (GC) and fine-grained locality through BSR and PDA, plus MLM. The model leverages an ESM-2 protein encoder and PubMedBERT biotext encoder to produce function-aware embeddings that excel across 22 benchmarks, achieving state-of-the-art results in protein classification, mutation effects, cross-modal transformation, semantic similarity, and PPI prediction. Key innovations include explicit modeling of static and dynamic functional segments and a memory-prototype-guided alignment, yielding substantial improvements over baselines (e.g., up to 75% in cross-modal transformation and notable GO term gains). Overall, ProtCLIP demonstrates that function-informed, data-efficient multi-modal learning can unlock robust, fine-grained protein representations with broad biological applicability and potential for guided protein design.

Abstract

Multi-modality pre-training paradigm that aligns protein sequences and biological descriptions has learned general protein representations and achieved promising performance in various downstream applications. However, these works were still unable to replicate the extraordinary success of language-supervised visual foundation models due to the ineffective usage of aligned protein-text paired data and the lack of an effective function-informed pre-training paradigm. To address these issues, this paper curates a large-scale protein-text paired dataset called ProtAnno with a property-driven sampling strategy, and introduces a novel function-informed protein pre-training paradigm. Specifically, the sampling strategy determines selecting probability based on the sample confidence and property coverage, balancing the data quality and data quantity in face of large-scale noisy data. Furthermore, motivated by significance of the protein specific functional mechanism, the proposed paradigm explicitly model protein static and dynamic functional segments by two segment-wise pre-training objectives, injecting fine-grained information in a function-informed manner. Leveraging all these innovations, we develop ProtCLIP, a multi-modality foundation model that comprehensively represents function-aware protein embeddings. On 22 different protein benchmarks within 5 types, including protein functionality classification, mutation effect prediction, cross-modal transformation, semantic similarity inference and protein-protein interaction prediction, our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks, 59.9% in GO-CC and 39.7% in GO-BP protein function prediction. The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.

Figures (7)

• Figure 1: Data distribution of ProtAnno-S and ProtAnno-D with different property coverage.
• Figure 2: Comparison results on 22 downstream benchmarks within five types. ProtCLIP consistently achieves the state-of-the-art performance on all these tasks. PPI: protein-protein interaction.
• Figure 3: Overview of ProtCLIP. We curate a large-scale protein-biotext dataset ProtAnno with a property-driven sampling strategy, and proposes a function-informed pre-training paradigm containing two segment-wise objectives BSR and PDA.
• Figure 4: An overview of downstream tasks within five types.
• Figure 5: Mutual interference between segment-wise reconstruction $\textbf{L}_\text{BSR}$ and token-wise reconstruction $\textbf{L}_\text{MLM}$ during the pre-training process.