Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leak Proof PDBBind: A Reorganized Dataset of Protein-Ligand Complexes for More Generalizable Binding Affinity Prediction

Jie Li, Xingyi Guan, Oufan Zhang, Kunyang Sun, Yingze Wang, Dorian Bagni, Teresa Head-Gordon

TL;DR

This work addresses the pervasive data leakage in PDBBind-based scoring-function evaluation by introducing LP-PDBBind, a leak-proof split that minimizes protein and ligand similarity between train, validation, and test sets. It retrains representative scoring functions (AutoDock Vina, RF-Score, IGN, DeepDTA) on LP-PDBBind and validates generalization using an independent BindingDB-derived benchmark (BDB2020+), plus SARS-CoV-2 Mpro and EGFR benchmark sets. The results show that LP-PDBBind improves transferability and that IGN, in particular, offers robust scoring and ranking across diverse protein–ligand systems, with gains extending to docked-pose rescoring. The authors advocate adopting independent benchmarks like BDB2020+ and report that their approach yields more realistic assessments of model performance, guiding the development of more generalizable binding-affinity predictors.

Abstract

The majority of machine learning scoring functions used in drug discovery for predicting protein-ligand binding poses and affinities have been trained on the PDBBind dataset. However, it is unclear whether these new scoring functions are actually an improvement over traditional models since often the training and test sets are cross-contaminated with proteins and ligands with high similarity, and hence they may not perform comparably well in binding prediction of unrelated protein-ligand complexes. In this work we have carefully prepared a new split of the PDBBind data set to control for data leakage, defined as proteins and ligands with high sequence and structural similarity. The resulting leak-proof (LP)-PDBBind data is used to retrain four popular SFs: AutoDock Vina, Random Forest (RF)-Score, InteractionGraphNet (IGN), and DeepDTA, to better test their capabilities when applied to new protein-ligand complexes. In particular we have formulated a new independent data set, BDB2020+, by matching high quality binding free energies from BindingDB with co-crystalized ligand-protein complexes from the PDB that have been deposited since 2020. Based on all the benchmark results, the retrained models using LP-PDBBind consistently perform better, with IGN especially being recommended for scoring and ranking applications for new protein-ligand systems.

Leak Proof PDBBind: A Reorganized Dataset of Protein-Ligand Complexes for More Generalizable Binding Affinity Prediction

TL;DR

This work addresses the pervasive data leakage in PDBBind-based scoring-function evaluation by introducing LP-PDBBind, a leak-proof split that minimizes protein and ligand similarity between train, validation, and test sets. It retrains representative scoring functions (AutoDock Vina, RF-Score, IGN, DeepDTA) on LP-PDBBind and validates generalization using an independent BindingDB-derived benchmark (BDB2020+), plus SARS-CoV-2 Mpro and EGFR benchmark sets. The results show that LP-PDBBind improves transferability and that IGN, in particular, offers robust scoring and ranking across diverse protein–ligand systems, with gains extending to docked-pose rescoring. The authors advocate adopting independent benchmarks like BDB2020+ and report that their approach yields more realistic assessments of model performance, guiding the development of more generalizable binding-affinity predictors.

Abstract

The majority of machine learning scoring functions used in drug discovery for predicting protein-ligand binding poses and affinities have been trained on the PDBBind dataset. However, it is unclear whether these new scoring functions are actually an improvement over traditional models since often the training and test sets are cross-contaminated with proteins and ligands with high similarity, and hence they may not perform comparably well in binding prediction of unrelated protein-ligand complexes. In this work we have carefully prepared a new split of the PDBBind data set to control for data leakage, defined as proteins and ligands with high sequence and structural similarity. The resulting leak-proof (LP)-PDBBind data is used to retrain four popular SFs: AutoDock Vina, Random Forest (RF)-Score, InteractionGraphNet (IGN), and DeepDTA, to better test their capabilities when applied to new protein-ligand complexes. In particular we have formulated a new independent data set, BDB2020+, by matching high quality binding free energies from BindingDB with co-crystalized ligand-protein complexes from the PDB that have been deposited since 2020. Based on all the benchmark results, the retrained models using LP-PDBBind consistently perform better, with IGN especially being recommended for scoring and ranking applications for new protein-ligand systems.
Paper Structure (15 sections, 1 equation, 4 figures, 4 tables)

This paper contains 15 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Data cleaning of PDBBind v2020
  4. New splitting of PDBBind dataset
  5. Retrained Models using the new split
  6. Compilation of the BDB2020+ dataset
  7. Preparation of two target-specific datasets
  8. Retraining Scoring Functions with LP-PDBBind
  9. Results
  10. Analysis of PDBBind Splittings
  11. Evaluating Retrained Models with LP-PDBBind
  12. Evaluating the Ranking Capabilities of the Retrained Models
  13. Rescoring Docked Poses and Pose Selection using the Retrained Scoring Functions
  14. Discussion and Conclusion
  15. Data Availability and Implementation

Figures (4)

  • Figure 1: Data statistics under different splits of the PDBBind dataset. Number of data in different splits of the PDBBind dataset: (a) general, refined and core set in original PDBBind split; (b) train, validation, test set and discarded data in Equibind split; (c) train, validation, test set and discarded data in LP-PDBBind. Comparison of maximum protein sequence similarities between test set and train set (or core set and general set in original PDBBind split) (d) or between validation set and train set under Equibind split and LP-PDBBind(e). Comparison of maximum ligand similarities between test set and train set (or core set and general set in original PDBBind split) (f) or between validation set and train set under Equibind split and LP-PDBBind(g). Comparison of maximum interaction fingerprint similarities between test set and train set (or core set and general set in original PDBBind split) (h) or between validation set and train set under Equibind split and LP-PDBBind(i).
  • Figure 2: Performance comparisons using different models and different benchmark datasets. (a) Comparison on the root mean square error (RMSE) for different models. Lower is better. Blue bars indicate RMSEs on the LP-PDBBind test dataset using retrained models, orange bars indicate RMSEs for the models without retraining using LP-PDBBind, and green bars indicate RMSEs for the models retrained using LP-PDBBind. (b) Comparison on the Pearson correlation coefficient ($R$) for different models.
  • Figure 3: Data statistics for the SARS-CoV-2 main protease (Mpro) benchmark set and epidermal growth factor receptor (EGFR) benchmark set. (a) Distributions of the binding affinity data ($-\log K_d$) in the LP-PDBBind train dataset in blue, Mpro benchmark set in orange and EGFR set in green. (b-c) Distributions of protein sequence similarities between the Mpro protein (b) and EGFR protein (c) with proteins in the LP-PDBBind train dataset. (d-e) Distributions of ligand fingerprint similarities between molecules in the Mpro benchmark set (d) and EGFR benchmark set (e) with ligands in the LP-PDBBind train dataset.
  • Figure 4: TOC graphic