Rethinking the generalization of drug target affinity prediction algorithms via similarity aware evaluation

TL;DR

The paper addresses the problem that standard randomized test splits in drug-target affinity prediction overstate generalization by overrepresenting high-similarity samples. It introduces Similarity Aware Evaluation (SAE), a differentiable framework that relaxes the test/train partition into a weighted split and optimizes a chi-square-like objective to achieve a user-specified similarity distribution across test samples, using LogSumExp proxies and softbinning with entropy regularization. SAE is demonstrated across four distributions (including balanced and mimic splits) and five representative DTA methods on multiple datasets, revealing that model performance correlates with similarity and that mimic splits can guide hyperparameter selection to align internal validation with external performance. The framework offers a flexible, scalable approach to generate meaningful test distributions for robust model development and has potential applications in QSAR, PPI, and DDI prediction, with code available at the provided repository.

Abstract

Drug-target binding affinity prediction is a fundamental task for drug discovery. It has been extensively explored in literature and promising results are reported. However, in this paper, we demonstrate that the results may be misleading and cannot be well generalized to real practice. The core observation is that the canonical randomized split of a test set in conventional evaluation leaves the test set dominated by samples with high similarity to the training set. The performance of models is severely degraded on samples with lower similarity to the training set but the drawback is highly overlooked in current evaluation. As a result, the performance can hardly be trusted when the model meets low-similarity samples in real practice. To address this problem, we propose a framework of similarity aware evaluation in which a novel split methodology is proposed to adapt to any desired distribution. This is achieved by a formulation of optimization problems which are approximately and efficiently solved by gradient descent. We perform extensive experiments across five representative methods in four datasets for two typical target evaluations and compare them with various counterpart methods. Results demonstrate that the proposed split methodology can significantly better fit desired distributions and guide the development of models. Code is released at https://github.com/Amshoreline/SAE/tree/main.

Figures (8)

• Figure 1: Comparison of randomized split and SAE (balanced) split at IC50 for EGFR. The randomized split led to 88% of test samples yielding a high similarity ($> 2/3$) to the training set. In contrast, our SAE (balanced) split strategy ensures a more balanced distribution of similarities. The evaluation of five DTA prediction methods demonstrates that the performance aligns with the similarity levels. In the randomized split, the overall performance closely resembles that of high-similarity samples, thus failing to evaluate the performance when encountering low-similarity samples.
• Figure 2: Impact of the hyper-parameter $\beta$ on the approximation of the maximum function in Eq. \ref{['align:beta']}. To illustrate this impact, we consider a simplified scenario involving only two random variables: $X$ and $Y$. (a) $Z = Max(X, Y)$; (b-d) $Z = 1 / \beta \log (\exp(\beta X) + \exp(\beta Y))$. A larger value of $\beta$ results in a more accurate approximation, with $\beta = 100$ yielding an excellent result.
• Figure 3: Influence of the hyper-parameter $\sigma$ in Eq. \ref{['align:sigma']}. we analyze a specific case where $K = 3, b_k = k / 3, c_k = (2k - 1) / 6$. (a) $y = \mathbb{I}(b_1 < r \le b_2)$; (b-d) $y = \exp \left((-(r - c_2)^2 / (2\sigma^2)\right) / \sum_{k=1}^3 \exp \left(-(r - c_k)^2 / (2\sigma^2)\right)$. A decrease in the value of $\sigma$ leads to a more precise estimation, with $\sigma = 0.01$ producing an outstanding result.
• Figure 4: The similarity distribution of the internal test set across different split strategies. (b) Randomized split leads to a scenario where most internal test samples are highly similar to the training set. (c) Scaffold split produces a more balanced distribution. (d) SIMPD split yields a distribution similar to the random split. (e) Our SAE (mimic) split brings the internal test set's distribution closest to that of the external test set.
• Figure 5: Comparison of the generalization ability of different split strategies at IC50 for EGFR across five DTA prediction methods. The external test set performance of the mimic split surpasses that of other split strategies.