Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing Reusability of Deep Learning-Based Monotherapy Drug Response Prediction Models Trained with Omics Data

Jamie C. Overbeek, Alexander Partin, Thomas S. Brettin, Nicholas Chia, Oleksandr Narykov, Priyanka Vasanthakumari, Andreas Wilke, Yitan Zhu, Austin Clyde, Sara Jones, Rohan Gnanaolivu, Yuanhang Liu, Jun Jiang, Chen Wang, Carter Knutson, Andrew McNaughton, Neeraj Kumar, Gayara Demini Fernando, Souparno Ghosh, Cesar Sanchez-Villalobos, Ruibo Zhang, Ranadip Pal, M. Ryan Weil, Rick L. Stevens

TL;DR

The paper tackles the reusability and reproducibility of deep learning–based drug response prediction models trained on omics data by introducing a structured scoring system across software environment, code modularity, and data preprocessing. It applies this framework to 17 models from the IMPROVE project, revealing pervasive gaps in environment specification, workflow documentation, and data preprocessing, while also attempting to reproduce key metrics to gauge practical adaptability. The authors provide concrete recommendations—emphasizing containerization, accessible preprocessing pipelines, modular documentation, and fresh-environment testing—to improve sharing practices without imposing heavy burdens on researchers. This work constitutes the first comprehensive assessment of DRP model reusability, highlighting the need for standardized guidelines to enable robust cross-model evaluation and accelerate clinical translation of AI-enabled oncology predictions.

Abstract

Cancer drug response prediction (DRP) models present a promising approach towards precision oncology, tailoring treatments to individual patient profiles. While deep learning (DL) methods have shown great potential in this area, models that can be successfully translated into clinical practice and shed light on the molecular mechanisms underlying treatment response will likely emerge from collaborative research efforts. This highlights the need for reusable and adaptable models that can be improved and tested by the wider scientific community. In this study, we present a scoring system for assessing the reusability of prediction DRP models, and apply it to 17 peer-reviewed DL-based DRP models. As part of the IMPROVE (Innovative Methodologies and New Data for Predictive Oncology Model Evaluation) project, which aims to develop methods for systematic evaluation and comparison DL models across scientific domains, we analyzed these 17 DRP models focusing on three key categories: software environment, code modularity, and data availability and preprocessing. While not the primary focus, we also attempted to reproduce key performance metrics to verify model behavior and adaptability. Our assessment of 17 DRP models reveals both strengths and shortcomings in model reusability. To promote rigorous practices and open-source sharing, we offer recommendations for developing and sharing prediction models. Following these recommendations can address many of the issues identified in this study, improving model reusability without adding significant burdens on researchers. This work offers the first comprehensive assessment of reusability and reproducibility across diverse DRP models, providing insights into current model sharing practices and promoting standards within the DRP and broader AI-enabled scientific research community.

Assessing Reusability of Deep Learning-Based Monotherapy Drug Response Prediction Models Trained with Omics Data

TL;DR

The paper tackles the reusability and reproducibility of deep learning–based drug response prediction models trained on omics data by introducing a structured scoring system across software environment, code modularity, and data preprocessing. It applies this framework to 17 models from the IMPROVE project, revealing pervasive gaps in environment specification, workflow documentation, and data preprocessing, while also attempting to reproduce key metrics to gauge practical adaptability. The authors provide concrete recommendations—emphasizing containerization, accessible preprocessing pipelines, modular documentation, and fresh-environment testing—to improve sharing practices without imposing heavy burdens on researchers. This work constitutes the first comprehensive assessment of DRP model reusability, highlighting the need for standardized guidelines to enable robust cross-model evaluation and accelerate clinical translation of AI-enabled oncology predictions.

Abstract

Cancer drug response prediction (DRP) models present a promising approach towards precision oncology, tailoring treatments to individual patient profiles. While deep learning (DL) methods have shown great potential in this area, models that can be successfully translated into clinical practice and shed light on the molecular mechanisms underlying treatment response will likely emerge from collaborative research efforts. This highlights the need for reusable and adaptable models that can be improved and tested by the wider scientific community. In this study, we present a scoring system for assessing the reusability of prediction DRP models, and apply it to 17 peer-reviewed DL-based DRP models. As part of the IMPROVE (Innovative Methodologies and New Data for Predictive Oncology Model Evaluation) project, which aims to develop methods for systematic evaluation and comparison DL models across scientific domains, we analyzed these 17 DRP models focusing on three key categories: software environment, code modularity, and data availability and preprocessing. While not the primary focus, we also attempted to reproduce key performance metrics to verify model behavior and adaptability. Our assessment of 17 DRP models reveals both strengths and shortcomings in model reusability. To promote rigorous practices and open-source sharing, we offer recommendations for developing and sharing prediction models. Following these recommendations can address many of the issues identified in this study, improving model reusability without adding significant burdens on researchers. This work offers the first comprehensive assessment of reusability and reproducibility across diverse DRP models, providing insights into current model sharing practices and promoting standards within the DRP and broader AI-enabled scientific research community.
Paper Structure (14 sections, 1 equation, 2 figures, 3 tables)

This paper contains 14 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Model selection criteria
  4. Software Environment
  5. Code Modularity
  6. Data Availability and Preprocessing
  7. Reproducing key performance scores
  8. Results
  9. Software Environment
  10. Code Modularity
  11. Data Availability and Preprocessing
  12. Reproducing key performance scores
  13. Discussion and Recommendations
  14. Conclusion

Figures (2)

  • Figure 1: Histograms showing the distribution of model scores for each category, with the number of models shown above each bar.
  • Figure 2: Percentage deviation of reproduced metrics, where the indicated metric is published. A negative score (red) means the reproduced value is lower than the published value. Pearson's correlation coefficient (PCC), coefficient of determination ($R^2$), area under the curve (AUC), and Spearman correlation coefficient (SCC) shown.