Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NovoBench: Benchmarking Deep Learning-based De Novo Peptide Sequencing Methods in Proteomics

Jingbo Zhou, Shaorong Chen, Jun Xia, Sizhe Liu, Tianze Ling, Wenjie Du, Yue Liu, Jianwei Yin, Stan Z. Li

TL;DR

This work presents the first unified benchmark NovoBench for peptide sequencing, which comprises diverse mass spectrum data, integrated models, and comprehensive evaluation metrics, and evaluates the models' performance in terms of identifying post-tranlational modifications (PTMs), efficiency and robustness to peptide length, noise peaks and missing fragment ratio.

Abstract

Tandem mass spectrometry has played a pivotal role in advancing proteomics, enabling the high-throughput analysis of protein composition in biological tissues. Many deep learning methods have been developed for \emph{de novo} peptide sequencing task, i.e., predicting the peptide sequence for the observed mass spectrum. However, two key challenges seriously hinder the further advancement of this important task. Firstly, since there is no consensus for the evaluation datasets, the empirical results in different research papers are often not comparable, leading to unfair comparison. Secondly, the current methods are usually limited to amino acid-level or peptide-level precision and recall metrics. In this work, we present the first unified benchmark NovoBench for \emph{de novo} peptide sequencing, which comprises diverse mass spectrum data, integrated models, and comprehensive evaluation metrics. Recent impressive methods, including DeepNovo, PointNovo, Casanovo, InstaNovo, AdaNovo and $π$-HelixNovo are integrated into our framework. In addition to amino acid-level and peptide-level precision and recall, we evaluate the models' performance in terms of identifying post-tranlational modifications (PTMs), efficiency and robustness to peptide length, noise peaks and missing fragment ratio, which are important influencing factors while seldom be considered. Leveraging this benchmark, we conduct a large-scale study of current methods, report many insightful findings that open up new possibilities for future development.

NovoBench: Benchmarking Deep Learning-based De Novo Peptide Sequencing Methods in Proteomics

TL;DR

This work presents the first unified benchmark NovoBench for peptide sequencing, which comprises diverse mass spectrum data, integrated models, and comprehensive evaluation metrics, and evaluates the models' performance in terms of identifying post-tranlational modifications (PTMs), efficiency and robustness to peptide length, noise peaks and missing fragment ratio.

Abstract

Tandem mass spectrometry has played a pivotal role in advancing proteomics, enabling the high-throughput analysis of protein composition in biological tissues. Many deep learning methods have been developed for \emph{de novo} peptide sequencing task, i.e., predicting the peptide sequence for the observed mass spectrum. However, two key challenges seriously hinder the further advancement of this important task. Firstly, since there is no consensus for the evaluation datasets, the empirical results in different research papers are often not comparable, leading to unfair comparison. Secondly, the current methods are usually limited to amino acid-level or peptide-level precision and recall metrics. In this work, we present the first unified benchmark NovoBench for \emph{de novo} peptide sequencing, which comprises diverse mass spectrum data, integrated models, and comprehensive evaluation metrics. Recent impressive methods, including DeepNovo, PointNovo, Casanovo, InstaNovo, AdaNovo and -HelixNovo are integrated into our framework. In addition to amino acid-level and peptide-level precision and recall, we evaluate the models' performance in terms of identifying post-tranlational modifications (PTMs), efficiency and robustness to peptide length, noise peaks and missing fragment ratio, which are important influencing factors while seldom be considered. Leveraging this benchmark, we conduct a large-scale study of current methods, report many insightful findings that open up new possibilities for future development.
Paper Structure (14 sections, 1 equation, 4 figures, 5 tables)

This paper contains 14 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Task Definition
  3. Datasets
  4. Baseline models
  5. Metrics
  6. Influencing Factors
  7. Results and Analysis
  8. Experimental Settings
  9. Main Results
  10. Influencing Factors
  11. Computational Efficiency
  12. Conclusion
  13. Limitation and Future Work
  14. Acknowledgements

Figures (4)

  • Figure 1: The overview of the NovoBench benchmark. The benchmark is organized incrementally from datasets to models, to metrics. We color contents in green and blue that are widely and partially considered by previous studies, respectively. Newly introduced contents are colored in pink.
  • Figure 2: Peptide-level precision curves of the benchmarking models under the influence of three factors on Nine-species dataset. The first row to the sixth row correspond to the models DeepNovo, PointNovo, CasaNovo, InstaNovo, AdaNovo, $\pi$-HelixNovo, and the first column to the third column correspond to the three influencing factors Peptide length, missing fragmentation ratio (MFR) and noise peaks (NSR). More results of the other two datasets can be found in the appendix.
  • Figure 3: Peptide-level precision curves of the benchmarking models under the influence of three factors on Seven-species dataset. The first row to the sixth row correspond to the models DeepNovo, PointNovo, CasaNovo, InstaNovo, AdaNovo, $\pi$-HelixNovo, and the first column to the third column correspond to the three influencing factors Peptide length, missing fragmentation ratio (MFR) and noise peaks (NSR).
  • Figure 4: Peptide-level precision curves of the benchmarking models under the influence of three factors on HC-PT dataset. The first row to the sixth row correspond to the models DeepNovo, PointNovo, CasaNovo, InstaNovo, AdaNovo, $\pi$-HelixNovo, and the first column to the third column correspond to the three influencing factors Peptide length, missing fragmentation ratio (MFR) and noise peaks (NSR).