Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment

TL;DR

Prot2Text-V2 introduces a decoder-only, multimodal framework that generates free-text protein function descriptions from amino acid sequences by aligning a pretrained protein encoder with a large language model via Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE). After a contrastive alignment stage, the decoder is instruction-tuned with LoRA to produce accurate, context-rich descriptions conditioned on sequence features, with robust performance in low-homology scenarios on a SwissProt-derived dataset. The approach achieves state-of-the-art lexical and semantic metrics, outperforming traditional methods and general-purpose LLMs, and demonstrates strong generalization where sequence similarity is limited. This work advances scalable, flexible protein annotation by enabling natural-language functional descriptions without heavy cross-attention modules and with efficient training and inference.

Abstract

Predicting protein function from sequence is a central challenge in computational biology. While existing methods rely heavily on structured ontologies or similarity-based techniques, they often lack the flexibility to express structure-free functional descriptions and novel biological functions. In this work, we introduce Prot2Text-V2, a novel multimodal sequence-to-text model that generates free-form natural language descriptions of protein function directly from amino acid sequences. Our method combines a protein language model as a sequence encoder (ESM-3B) and a decoder-only language model (LLaMA-3.1-8B-Instruct) through a lightweight nonlinear modality projector. A key innovation is our Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE), which improves cross-modal learning by matching mean- and std-pooled protein embeddings with text representations via contrastive loss. After the alignment phase, we apply instruction-based fine-tuning using LoRA on the decoder to teach the model how to generate accurate protein function descriptions conditioned on the protein sequence. We train Prot2Text-V2 on about 250K curated entries from SwissProt and evaluate it under low-homology conditions, where test sequences have low similarity with training samples. Prot2Text-V2 consistently outperforms traditional and LLM-based baselines across various metrics.

Figures (11)

• Figure 1: Illustration of the multimodal architecture and primary data flow of Prot2Text-V2. (a) In the protein encoder, the amino-acid sequence is first encoded by the protein language model and then projected to the hidden dimension of the text decoder with a two-layer non-linear modality projector. (b) In the language model decoder, the projected sequence of protein embeddings is inserted into the sequence of word embeddings of the user message, forming the prompt to the language model. Finally, the decoder generates the description of the protein auto-regressively.
• Figure 2: A flexible chat template for language decoders, where user message metadata fields are subject to dropout during training. During evaluation, these fields are either fully retained or completely removed, enabling two distinct testing scenarios.
• Figure 3: Illustration of the two-stage training process of Prot2Text-V2. (a) During the first stage, Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE) aligns the protein encoder outputs with the semantic space of the pretrained language decoder, enabling effective cross-modal learning. (b) In the second stage, instruction-based supervised fine-tuning is performed to teach the decoder how to generate protein function descriptions based on the aligned embeddings. Red blocks denote trainable parameters, while blue blocks indicate frozen components.
• Figure 4: Illustration on similarity matrices before (left) and after (right) alignment on a group of test protein-description pairs. After alignment, protein embeddings show strong one-to-one correspondence with their matching text embeddings, as indicated by the prominent diagonal. This reflects significantly higher similarity between each protein sequence embedding and its own text description compared to mismatched pairs, demonstrating successful cross-modal alignment.
• Figure 5: Evaluation across BLAST sequence bitscore bins (%) shows that Prot2Text-V2 outperform baselines (Prot2Text and BLAST). The evaluations show consistent improvements, particularly in the [0–50] bins, highlighting better generalization to remote homologs.