Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking drug-drug interaction prediction methods: a perspective of distribution changes

Zhenqian Shen, Mingyang Zhou, Yongqi Zhang, Quanming Yao

TL;DR

Emerging drug–drug interaction prediction is hindered by distribution changes between known and new drugs, which are not captured by traditional i.i.d. evaluations. The paper introduces DDI-Ben, a benchmarking framework with a distribution-change surrogate and cluster-based drug splits that reflect real-world drug development, and benchmarks ten diverse methods on DrugBank and TWOSIDES. Results show substantial performance degradation under distribution changes for most methods, while LLM-based approaches (e.g., DDI-GPT, TextDDI) leveraging textual drug information exhibit greater robustness. The work provides benchmark datasets with simulated distribution changes and highlights directions for future research, including language-model–driven methods and domain adaptation to improve resilience in emerging DDI prediction.

Abstract

Motivation: Emerging drug-drug interaction (DDI) prediction is crucial for new drugs but is hindered by distribution changes between known and new drugs in real-world scenarios. Current evaluation often neglects these changes, relying on unrealistic i.i.d. split due to the absence of drug approval data. Results: We propose DDI-Ben, a benchmarking framework for emerging DDI prediction under distribution changes. DDI-Ben introduces a distribution change simulation framework that leverages distribution changes between drug sets as a surrogate for real-world distribution changes of DDIs, and is compatible with various drug split strategies. Through extensive benchmarking on ten representative methods, we show that most existing approaches suffer substantial performance degradation under distribution changes. Our analysis further indicates that large language model (LLM) based methods and the integration of drug-related textual information offer promising robustness against such degradation. To support future research, we release the benchmark datasets with simulated distribution changes. Overall, DDI-Ben highlights the importance of explicitly addressing distribution changes and provides a foundation for developing more resilient methods for emerging DDI prediction. Availability and implementation: Our code and data are available at https://github.com/LARS-research/DDI-Bench.

Benchmarking drug-drug interaction prediction methods: a perspective of distribution changes

TL;DR

Emerging drug–drug interaction prediction is hindered by distribution changes between known and new drugs, which are not captured by traditional i.i.d. evaluations. The paper introduces DDI-Ben, a benchmarking framework with a distribution-change surrogate and cluster-based drug splits that reflect real-world drug development, and benchmarks ten diverse methods on DrugBank and TWOSIDES. Results show substantial performance degradation under distribution changes for most methods, while LLM-based approaches (e.g., DDI-GPT, TextDDI) leveraging textual drug information exhibit greater robustness. The work provides benchmark datasets with simulated distribution changes and highlights directions for future research, including language-model–driven methods and domain adaptation to improve resilience in emerging DDI prediction.

Abstract

Motivation: Emerging drug-drug interaction (DDI) prediction is crucial for new drugs but is hindered by distribution changes between known and new drugs in real-world scenarios. Current evaluation often neglects these changes, relying on unrealistic i.i.d. split due to the absence of drug approval data. Results: We propose DDI-Ben, a benchmarking framework for emerging DDI prediction under distribution changes. DDI-Ben introduces a distribution change simulation framework that leverages distribution changes between drug sets as a surrogate for real-world distribution changes of DDIs, and is compatible with various drug split strategies. Through extensive benchmarking on ten representative methods, we show that most existing approaches suffer substantial performance degradation under distribution changes. Our analysis further indicates that large language model (LLM) based methods and the integration of drug-related textual information offer promising robustness against such degradation. To support future research, we release the benchmark datasets with simulated distribution changes. Overall, DDI-Ben highlights the importance of explicitly addressing distribution changes and provides a foundation for developing more resilient methods for emerging DDI prediction. Availability and implementation: Our code and data are available at https://github.com/LARS-research/DDI-Bench.

Paper Structure

This paper contains 38 sections, 1 equation, 15 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Emerging DDI Prediction Task Description
  3. Distribution Change Simulation Framework
  4. Drug Distribution Change as Surrogate
  5. Understanding
  6. Customized Distribution Change Surrogate
  7. Comparison among Different Data Split Strategies
  8. Instantiation of the Proposed Framework
  9. Dataset and Split
  10. Methods to be Compared
  11. Evaluation Metric
  12. Empirical Results
  13. Performance Analysis among Representative Methods
  14. Revealing Factors that can Alleviate the Negative Impact of Distribution Change
  15. Performance Analysis for Different DDI Types
  16. ...and 23 more sections

Figures (15)

  • Figure 1: Comparison between common DDI data split and proposed distribution change simulation framework for emerging DDI prediction evaluation.
  • Figure 2: Emerging DDI prediction task description.
  • Figure 3: Illustration of DDI data split.
  • Figure 4: T-SNE illustration of drug distribution with their approval time in Drugbank dataset.
  • Figure 5: Figure for time threshold of the realistic drug split scheme w.r.t the consistency index of each drug split scheme.
  • ...and 10 more figures

Theorems & Definitions (1)

  • Remark 1