Protein Representation Learning by Capturing Protein Sequence-Structure-Function Relationship

TL;DR

This work tackles the challenge of learning comprehensive protein representations by jointly modeling sequence, structure, and function. It introduces AMMA, a masked autoencoder with a unified multi-modal encoder and asymmetric decoders to capture the asymmetric relationships among modalities. Empirical results show that AMMA outperforms state-of-the-art baselines on protein function prediction and can further improve performance by leveraging unpaired data, all with lower pretraining resources than comparable models. The approach offers a practical and efficient path toward balanced, multi-modal protein representations with broad downstream applicability.

Abstract

The goal of protein representation learning is to extract knowledge from protein databases that can be applied to various protein-related downstream tasks. Although protein sequence, structure, and function are the three key modalities for a comprehensive understanding of proteins, existing methods for protein representation learning have utilized only one or two of these modalities due to the difficulty of capturing the asymmetric interrelationships between them. To account for this asymmetry, we introduce our novel asymmetric multi-modal masked autoencoder (AMMA). AMMA adopts (1) a unified multi-modal encoder to integrate all three modalities into a unified representation space and (2) asymmetric decoders to ensure that sequence latent features reflect structural and functional information. The experiments demonstrate that the proposed AMMA is highly effective in learning protein representations that exhibit well-aligned inter-modal relationships, which in turn makes it effective for various downstream protein-related tasks.

Figures (8)

• Figure 1: t-SNE visualization of the three modalities of proteins. Latent features for sequence (red), structure (yellow), and function (blue) extracted from ESM-1b, GearNet, and PubMedBERT-abs, respectively, are visualized. Two proteins, 30S ribosomal protein S13 (S) and 50S ribosomal protein L22 (L), are functionally similar and therefore proximal in function space (left). These proteins are encoded close together in structure space, but far apart in sequence space (middle). This trend is common across ribosomal proteins (right). Details are provided in Section \ref{['sec:tsne_detail']}.
• Figure 2: Multi-modal protein representation learning with AMMA. AMMA has two key components: (a) a unified multi-modal encoder and (b) asymmetric decoders. Each modality is encoded by a frozen pretrained encoder, then integrated by a multi-modal encoder after masking. Asymmetric decoders then reconstruct original features of each modality. During decoding, the input latent features, designed to hold target-specific information, are asymmetrically passed to the decoders for target modality reconstruction. This requires AMMA to encode structural and functional information into sequence latent features, which allows AMMA to capture unique asymmetric sequence-structure-function relationships. The overall architecture is provided in Figure \ref{['fig:amma']}.
• Figure 3: EC/GO results of 15 epochs with extra unpaired data.
• Figure 3: Visualization of highly attended residues in a functional context.
• Figure 4: t-SNE visualization of the three protein modalities after (a) AMMA training and (b) contrastive learning. Sequence, structure, and function features are well-aligned after training with AMMA while contrastive learning fails to align the three modalities in a balanced manner. Details are provided in Section \ref{['sec:tsne_detail']}.