Deep-Learning Based Docking Methods: Fair Comparisons to Conventional Docking Workflows

TL;DR

This work challenges the claim that DiffDock provides a superior docking approach by establishing a fair baseline with fully automatic conventional docking workflows. It shows that mature methods such as Surflex-Dock and Glide outperform DiffDock under known-binding-site conditions, and that even in blind docking, conventional baselines remain stronger, highlighting critical issues in training-test splits and near-neighbor contamination. The analysis demonstrates that much of DiffDock’s reported success arises from memorization of training data rather than true docking capability, urging rigorous benchmarking and cautious interpretation of AI-based docking claims. The Addendum acknowledges later DiffDock variants and provides resources to support reproducibility and ongoing evaluation in the field.

Abstract

The diffusion learning method, DiffDock, for docking small-molecule ligands into protein binding sites was recently introduced. Results included comparisons to more conventional docking approaches, with DiffDock showing superior performance. Here, we employ a fully automatic workflow using the Surflex-Dock methods to generate a fair baseline for conventional docking approaches. Results were generated for the common and expected situation where a binding site location is known and also for the condition of an unknown binding site. For the known binding site condition, Surflex-Dock success rates at 2.0 Angstroms RMSD far exceeded those for DiffDock (Top-1/Top-5 success rates, respectively, were 68/81% compared with 45/51%). Glide performed with similar success rates (67/73%) to Surflex-Dock for the known binding site condition, and results for AutoDock Vina and Gnina followed this pattern. For the unknown binding site condition, using an automated method to identify multiple binding pockets, Surflex-Dock success rates again exceeded those of DiffDock, but by a somewhat lesser margin. DiffDock made use of roughly 17,000 co-crystal structures for learning (98% of PDBBind version 2020, pre-2019 structures) for a training set in order to predict on 363 test cases (2% of PDBBind 2020) from 2019 forward. DiffDock's performance was inextricably linked with the presence of near-neighbor cases of close to identical protein-ligand complexes in the training set for over half of the test set cases. DiffDock exhibited a 40 percentage point difference on near-neighbor cases (two-thirds of all test cases) compared with cases with no near-neighbor training case. DiffDock has apparently encoded a type of table-lookup during its learning process, rendering meaningful applications beyond its reach. Further, it does not perform even close to competitively with a competently run modern docking workflow.

Figures (7)

• Figure 1: Cumulative histogram of RMSD values for Top-1 and Top-5 DiffDock predicted poses for the Full Test Set (cyan and yellow curves) and the Clean Test Set from full PDB re-processing (violet and green curves).
• Figure 2: Cumulative histogram of RMSD values for Top-1 and Top-5 Surflex-Dock (cyan and yellow) and DiffDock poses (violet and green) for the Clean Test Set with known binding sites (left) and with unknown binding sites (right).
• Figure 3: Cumulative histogram of RMSD values for Top-1 and Top-5 Glide (cyan and yellow) and DiffDock poses (violet and green) for the Clean Test Set with known binding sites.
• Figure 4: Cumulative histograms of RMSD values for Top-1 and Top-5 Vina and Gnina (cyan and yellow, left and right, respectively) and DiffDock poses (violet and green) for the Clean Test Set with known binding sites.
• Figure 5: Examples of near-neighbor success cases for DiffDock: HIV-protease (top) and BACE1 (bottom).