Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis

TL;DR

The paper tackles the challenge of assessing cross-dataset generalization in drug response prediction (DRP) by introducing a standardized benchmarking framework that combines five public drug-screening datasets, six standardized models (five deep-learning DRP models plus LGBM), and a scalable cross-dataset workflow. It defines four metrics ($G$, $G_a$, $G_n$, $G_{na}$) to quantify absolute and relative transferability across datasets, enabling fair and systematic comparisons. Empirical results reveal substantial declines in cross-dataset performance relative to within-dataset performance, though certain model-dataset pairings (notably UNO, GraphDRP, LGBM) show partial robustness and CTRPv2 stands out as a strong training source. The framework and metrics provide a rigorous, reusable foundation for evaluating and improving DRP models toward real-world generalization and clinical relevance.

Abstract

Deep learning (DL) and machine learning (ML) models have shown promise in drug response prediction (DRP), yet their ability to generalize across datasets remains an open question, raising concerns about their real-world applicability. Due to the lack of standardized benchmarking approaches, model evaluations and comparisons often rely on inconsistent datasets and evaluation criteria, making it difficult to assess true predictive capabilities. In this work, we introduce a benchmarking framework for evaluating cross-dataset prediction generalization in DRP models. Our framework incorporates five publicly available drug screening datasets, six standardized DRP models, and a scalable workflow for systematic evaluation. To assess model generalization, we introduce a set of evaluation metrics that quantify both absolute performance (e.g., predictive accuracy across datasets) and relative performance (e.g., performance drop compared to within-dataset results), enabling a more comprehensive assessment of model transferability. Our results reveal substantial performance drops when models are tested on unseen datasets, underscoring the importance of rigorous generalization assessments. While several models demonstrate relatively strong cross-dataset generalization, no single model consistently outperforms across all datasets. Furthermore, we identify CTRPv2 as the most effective source dataset for training, yielding higher generalization scores across target datasets. By sharing this standardized evaluation framework with the community, our study aims to establish a rigorous foundation for model comparison, and accelerate the development of robust DRP models for real-world applications.

Figures (9)

• Figure 1: Basic components in the development of drug response prediction (DRP) models. The process consists of three main stages: (1) Data Preparation: including benchmark data components (e.g., drug response, omics, and drug feature data) and supplementary model-specific data; (2) Model Development: a general ML pipeline structured into three distinct stages—preprocessing, training, and inference; (3) Performance Evaluation: cross-dataset generalization assessment (the example $G$ matrix is shown for illustration purposes and does not represent real experimental results).
• Figure 2: The main data components in the drug response benchmark dataset and their integration through shared cell and drug identifiers.
• Figure 3: General machine learning (ML) pipeline. This includes preprocessing, training, and inference. The dashed lines signify optional elements (Supp. Data refers to data utilized by the model that is not part of the benchmark dataset).
• Figure 4: Cross-Dataset Generalization. A. Computing a prediction performance score for the $G[CCLE, CCLE]$ matrix entry (within-study case). B. Computing a prediction performance score for the $G[CCLE, gCSI]$ matrix entry (cross-dataset case). The parallel implementation of this workflow is implemented using Parsl.
• Figure 5: Cross-dataset performance matrices for each model. Each pair includes the basic matrix $G$\ref{['sec:G_matrix']} (blue) and the normalized matrix $G_n$\ref{['sec:Gn_matrix']} (green). In the $G$ matrices, values represent mean $R^2$ scores across splits, and the numbers in parentheses indicate standard deviations.