Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty Quantification on Clinical Trial Outcome Prediction

Tianyi Chen, Yingzhou Lu, Nan Hao, Yuanyuan Zhang, Capucine Van Rechem, Jintai Chen, Tianfan Fu

TL;DR

This work tackles uncertainty in clinical trial outcome prediction by integrating selective classification with the Hierarchical Interaction Network (HINT). The approach uses multi-modal input embeddings, external pharmaco-kinetics knowledge, and a four-tier hierarchical interaction graph to produce trial approval probabilities while abstaining on low-confidence cases. Across Phase I–III predictions on a public TOP dataset, the method achieves substantial improvements in PR-AUC (up to 32.37% relative in Phase I) and reaches a PR-AUC of 0.9022 in Phase III, demonstrating robustness across diseases and phases. The study highlights the practical impact of uncertainty-aware predictions in speeding up drug development, reducing wasted resources, and guiding human interventions in high-stakes clinical decision-making.

Abstract

The importance of uncertainty quantification is increasingly recognized in the diverse field of machine learning. Accurately assessing model prediction uncertainty can help provide deeper understanding and confidence for researchers and practitioners. This is especially critical in medical diagnosis and drug discovery areas, where reliable predictions directly impact research quality and patient health. In this paper, we proposed incorporating uncertainty quantification into clinical trial outcome predictions. Our main goal is to enhance the model's ability to discern nuanced differences, thereby significantly improving its overall performance. We have adopted a selective classification approach to fulfill our objective, integrating it seamlessly with the Hierarchical Interaction Network (HINT), which is at the forefront of clinical trial prediction modeling. Selective classification, encompassing a spectrum of methods for uncertainty quantification, empowers the model to withhold decision-making in the face of samples marked by ambiguity or low confidence, thereby amplifying the accuracy of predictions for the instances it chooses to classify. A series of comprehensive experiments demonstrate that incorporating selective classification into clinical trial predictions markedly enhances the model's performance, as evidenced by significant upticks in pivotal metrics such as PR-AUC, F1, ROC-AUC, and overall accuracy. Specifically, the proposed method achieved 32.37\%, 21.43\%, and 13.27\% relative improvement on PR-AUC over the base model (HINT) in phase I, II, and III trial outcome prediction, respectively. When predicting phase III, our method reaches 0.9022 PR-AUC scores. These findings illustrate the robustness and prospective utility of this strategy within the area of clinical trial predictions, potentially setting a new benchmark in the field.

Uncertainty Quantification on Clinical Trial Outcome Prediction

TL;DR

This work tackles uncertainty in clinical trial outcome prediction by integrating selective classification with the Hierarchical Interaction Network (HINT). The approach uses multi-modal input embeddings, external pharmaco-kinetics knowledge, and a four-tier hierarchical interaction graph to produce trial approval probabilities while abstaining on low-confidence cases. Across Phase I–III predictions on a public TOP dataset, the method achieves substantial improvements in PR-AUC (up to 32.37% relative in Phase I) and reaches a PR-AUC of 0.9022 in Phase III, demonstrating robustness across diseases and phases. The study highlights the practical impact of uncertainty-aware predictions in speeding up drug development, reducing wasted resources, and guiding human interventions in high-stakes clinical decision-making.

Abstract

The importance of uncertainty quantification is increasingly recognized in the diverse field of machine learning. Accurately assessing model prediction uncertainty can help provide deeper understanding and confidence for researchers and practitioners. This is especially critical in medical diagnosis and drug discovery areas, where reliable predictions directly impact research quality and patient health. In this paper, we proposed incorporating uncertainty quantification into clinical trial outcome predictions. Our main goal is to enhance the model's ability to discern nuanced differences, thereby significantly improving its overall performance. We have adopted a selective classification approach to fulfill our objective, integrating it seamlessly with the Hierarchical Interaction Network (HINT), which is at the forefront of clinical trial prediction modeling. Selective classification, encompassing a spectrum of methods for uncertainty quantification, empowers the model to withhold decision-making in the face of samples marked by ambiguity or low confidence, thereby amplifying the accuracy of predictions for the instances it chooses to classify. A series of comprehensive experiments demonstrate that incorporating selective classification into clinical trial predictions markedly enhances the model's performance, as evidenced by significant upticks in pivotal metrics such as PR-AUC, F1, ROC-AUC, and overall accuracy. Specifically, the proposed method achieved 32.37\%, 21.43\%, and 13.27\% relative improvement on PR-AUC over the base model (HINT) in phase I, II, and III trial outcome prediction, respectively. When predicting phase III, our method reaches 0.9022 PR-AUC scores. These findings illustrate the robustness and prospective utility of this strategy within the area of clinical trial predictions, potentially setting a new benchmark in the field.
Paper Structure (23 sections, 20 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 23 sections, 20 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Formulation
  4. Significance of Clinical Trial Outcome Prediction
  5. Base Model: HINT
  6. Input Embedding Module
  7. Pretraining Using External Knowledge
  8. Hierarchical Interaction Graph
  9. Selective Classification to Quantify Uncertainty
  10. Selective Classification (SC)
  11. Dataset
  12. Temporal data split based on start and completion date.
  13. Evaluation
  14. Results
  15. Quantitative Results
  16. ...and 8 more sections

Figures (5)

  • Figure 1: HINT extracts features from the following trial components: drug molecule embedding $\mathbf{e}_\tau$, disease embedding $\mathbf{e}_\delta$, and trial protocol embedding $\mathbf{e}_\pi$ (as described in section \ref{['sec:input']}). Before constructing an interaction graph using these components, HINT pretrains certain embeddings (depicted as blue nodes) using external knowledge about medication characteristics and disease risks(Section \ref{['sec:knowledge']}). Subsequently, we create an interaction graph in Section \ref{['sec:gnn']} to depict the interactions among different trial components. Using this interaction graph, we obtain trial embeddings that represent the trial components and their interactions. Leveraging the learned embeddings, we make predictions for trial approvals.
  • Figure 2: Selective Classification on HINT.
  • Figure 3: Dataset. The dataset is curated by aggregating multi-modal data from various sources. This dataset contains data on medications (drug molecules), diseases, trial protocols (text data), and approval information (labels).
  • Figure 4: Phase-Level outcome prediction (Average accuracy and standard deviation). Our method significantly outperforms HINT (i.e., passing hypothesis testing, the p-value is smaller than 0.05) in all the metrics in all the tasks.
  • Figure 5: Tradeoff between Selective Accuracy and Fraction Kept.

Theorems & Definitions (4)

  • Definition 1: Treatment Set
  • Definition 2: Target Disease Set
  • Definition 3: Trial Protocol
  • Definition 4: Clinical Trial Approval