SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation

TL;DR

A novel pre-training strategy for protein foundation models that emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features from sequence data is introduced.

Abstract

Proteins, essential to biological systems, perform functions intricately linked to their three-dimensional structures. Understanding the relationship between protein structures and their amino acid sequences remains a core challenge in protein modeling. While traditional protein foundation models benefit from pre-training on vast unlabeled datasets, they often struggle to capture critical co-evolutionary information, which evolutionary-based methods excel at. In this study, we introduce a novel pre-training strategy for protein foundation models that emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features from sequence data. Trained on a large-scale protein sequence dataset, our model demonstrates superior generalization ability, outperforming established baselines of similar size, including the ESM model, across diverse downstream tasks. Experimental results confirm the model's effectiveness in integrating co-evolutionary information, marking a significant step forward in protein sequence-based modeling.

Figures (3)

• Figure 1: Illustration of Local and Global Interactions. The local interactions are highlighted in red, while the global interactions are highlighted in cyan. (PDB ID: 4UQX)
• Figure 2: An overview of the SFM-Protein framework
• Figure 3: Loss vs. Compute during Pre-training