Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training

Le Zhuo, Zewen Chi, Minghao Xu, Heyan Huang, Heqi Zheng, Conghui He, Xian-Ling Mao, Wentao Zhang

TL;DR

ProtLLM presents a versatile cross-modal LLM capable of processing interleaved protein–text inputs by introducing dynamic protein mounting and a protein-as-word pre-training objective. It couples a protein encoder with an autoregressive language model via cross-modal connectors and trains on InterPT, a large-scale corpus of interleaved protein and text data, including multi-protein articles, protein annotations, and instruction-following data. The approach achieves competitive protein-centric performance against specialized baselines and unlocks zero-shot and in-context learning for protein-language tasks, including protein retrieval guided by textual prompts. The work demonstrates practical impact in tasks like enzyme mining and PPI prediction, highlighting ProtLLM’s potential to accelerate bioscience research through unified, flexible multi-protein understanding. Limitations point to extending modalities beyond sequences and exploring broader scientific discovery tasks.

Abstract

We propose ProtLLM, a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs where the natural language text is interspersed with an arbitrary number of proteins. Besides, we propose the protein-as-word language modeling approach to train ProtLLM. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates. Additionally, we construct a large-scale interleaved protein-text dataset, named InterPT, for pre-training. This dataset comprehensively encompasses both (1) structured data sources like protein annotations and (2) unstructured data sources like biological research papers, thereby endowing ProtLLM with crucial knowledge for understanding proteins. We evaluate ProtLLM on classic supervised protein-centric tasks and explore its novel protein-language applications. Experimental results demonstrate that ProtLLM not only achieves superior performance against protein-specialized baselines on protein-centric tasks but also induces zero-shot and in-context learning capabilities on protein-language tasks.

ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training

TL;DR

ProtLLM presents a versatile cross-modal LLM capable of processing interleaved protein–text inputs by introducing dynamic protein mounting and a protein-as-word pre-training objective. It couples a protein encoder with an autoregressive language model via cross-modal connectors and trains on InterPT, a large-scale corpus of interleaved protein and text data, including multi-protein articles, protein annotations, and instruction-following data. The approach achieves competitive protein-centric performance against specialized baselines and unlocks zero-shot and in-context learning for protein-language tasks, including protein retrieval guided by textual prompts. The work demonstrates practical impact in tasks like enzyme mining and PPI prediction, highlighting ProtLLM’s potential to accelerate bioscience research through unified, flexible multi-protein understanding. Limitations point to extending modalities beyond sequences and exploring broader scientific discovery tasks.

Abstract

We propose ProtLLM, a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs where the natural language text is interspersed with an arbitrary number of proteins. Besides, we propose the protein-as-word language modeling approach to train ProtLLM. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates. Additionally, we construct a large-scale interleaved protein-text dataset, named InterPT, for pre-training. This dataset comprehensively encompasses both (1) structured data sources like protein annotations and (2) unstructured data sources like biological research papers, thereby endowing ProtLLM with crucial knowledge for understanding proteins. We evaluate ProtLLM on classic supervised protein-centric tasks and explore its novel protein-language applications. Experimental results demonstrate that ProtLLM not only achieves superior performance against protein-specialized baselines on protein-centric tasks but also induces zero-shot and in-context learning capabilities on protein-language tasks.
Paper Structure (34 sections, 2 equations, 4 figures, 6 tables)

This paper contains 34 sections, 2 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models
  4. Protein Representation Learning
  5. Methods
  6. ProtLLM Framework
  7. Model architecture
  8. Dynamic protein mounting
  9. ProtLLM Pre-Training
  10. Protein-as-word language modeling
  11. Pre-training acceleration with protein cache
  12. InterPT: Interleaving Protein-Text Data
  13. Multi-protein scientific articles
  14. Protein-annotation pairs
  15. Protein instruction-following data
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Unlike existing protein representation models that focus on protein-text pairs or protein-only data, ProtLLM can handle complex inputs with multiple proteins interleaved with text, thereby learning crucial knowledge from scientific papers and supporting diverse downstream tasks.
  • Figure 2: An overview of ProtLLM. The architecture of ProtLLM consists of an autoregressive transformer, a protein encoder, and cross-modal connectors. With dynamic protein mounting, ProtLLM adeptly handles free-form interleaved protein-text sequences with an arbitrary number of proteins in the input. ProtLLM is pre-trained with protein-as-word language modeling that unifies word and protein prediction by constructing a protein vocabulary.
  • Figure 3: In-context learning results on human PPI.
  • Figure 4: Top-1 enzyme mining results based on ProtLLM retrieval and AutoDock Vina post-screening. $K_{cat} / K_M$ and $K_{cat}$ measure enzyme activity (higher the better). Vina energy measures binding affinity (lower the better).