Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProteinBench: A Holistic Evaluation of Protein Foundation Models

Fei Ye, Zaixiang Zheng, Dongyu Xue, Yuning Shen, Lihao Wang, Yiming Ma, Yan Wang, Xinyou Wang, Xiangxin Zhou, Quanquan Gu

TL;DR

ProteinBench tackles the fragmentation in evaluating protein foundation models by proposing a holistic benchmark that combines a taxonomy of protein tasks, a four-dimension multi-metric evaluation (quality, novelty, diversity, robustness), and deep-dive analyses from multiple user perspectives, all with an open leaderboard. The framework enables fair cross-task and cross-model comparisons across protein design and conformational dynamics, leveraging diverse datasets and state-of-the-art evaluation tools. Key findings show language-models excel at capturing native evolutionary distributions while structure-based approaches offer robustness for de novo design, yet no single method dominates across all tasks, highlighting the need for task-aligned model selection and potential integration of strengths. By providing public data, code, and a living leaderboard, ProteinBench aims to accelerate responsible development and practical deployment of protein foundation models in research and industry.

Abstract

Recent years have witnessed a surge in the development of protein foundation models, significantly improving performance in protein prediction and generative tasks ranging from 3D structure prediction and protein design to conformational dynamics. However, the capabilities and limitations associated with these models remain poorly understood due to the absence of a unified evaluation framework. To fill this gap, we introduce ProteinBench, a holistic evaluation framework designed to enhance the transparency of protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance. Our comprehensive evaluation of protein foundation models reveals several key findings that shed light on their current capabilities and limitations. To promote transparency and facilitate further research, we release the evaluation dataset, code, and a public leaderboard publicly for further analysis and a general modular toolkit. We intend for ProteinBench to be a living benchmark for establishing a standardized, in-depth evaluation framework for protein foundation models, driving their development and application while fostering collaboration within the field.

ProteinBench: A Holistic Evaluation of Protein Foundation Models

TL;DR

ProteinBench tackles the fragmentation in evaluating protein foundation models by proposing a holistic benchmark that combines a taxonomy of protein tasks, a four-dimension multi-metric evaluation (quality, novelty, diversity, robustness), and deep-dive analyses from multiple user perspectives, all with an open leaderboard. The framework enables fair cross-task and cross-model comparisons across protein design and conformational dynamics, leveraging diverse datasets and state-of-the-art evaluation tools. Key findings show language-models excel at capturing native evolutionary distributions while structure-based approaches offer robustness for de novo design, yet no single method dominates across all tasks, highlighting the need for task-aligned model selection and potential integration of strengths. By providing public data, code, and a living leaderboard, ProteinBench aims to accelerate responsible development and practical deployment of protein foundation models in research and industry.

Abstract

Recent years have witnessed a surge in the development of protein foundation models, significantly improving performance in protein prediction and generative tasks ranging from 3D structure prediction and protein design to conformational dynamics. However, the capabilities and limitations associated with these models remain poorly understood due to the absence of a unified evaluation framework. To fill this gap, we introduce ProteinBench, a holistic evaluation framework designed to enhance the transparency of protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance. Our comprehensive evaluation of protein foundation models reveals several key findings that shed light on their current capabilities and limitations. To promote transparency and facilitate further research, we release the evaluation dataset, code, and a public leaderboard publicly for further analysis and a general modular toolkit. We intend for ProteinBench to be a living benchmark for establishing a standardized, in-depth evaluation framework for protein foundation models, driving their development and application while fostering collaboration within the field.
Paper Structure (42 sections, 2 figures, 15 tables)

This paper contains 42 sections, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Background and task definition
  3. Protein Design
  4. Inverse folding
  5. Protein backbone design
  6. Protein sequence design
  7. Structure and sequence co-design
  8. Motif scaffolding
  9. Antibody Design
  10. Protein Conformation Prediction
  11. Protein Folding: single-state prediction
  12. Multiple-state prediction
  13. Distribution prediction
  14. ProteinBench
  15. Protein Design
  16. ...and 27 more sections

Figures (2)

  • Figure 1: Comprehensive overview of fundamental protein modeling tasks in ProteinBench. ProteinBench incorporates a spectrum of protein modeling challenges. Tasks marked with blue stars highlight domains where standardized performance benchmarks were previously unavailable.
  • Figure 2: Performance of motif-scaffolding of structure-based and sequence-based methods on the benchmark used in watson2023rfdiffusion. Results of FrameFlow, RFDiffusion and TDS are quoted from yim2024improved.