Automatically Labeling Clinical Trial Outcomes: A Large-Scale Benchmark for Drug Development

TL;DR

This paper introduces CTO, a large-scale, public benchmark and knowledge base for clinical trial outcomes, aggregating about 125,000 drug and biologic trials and integrating multimodal signals from trial publications (including LLM interpretations of PubMed abstracts), phase progression, news sentiment, sponsor stock prices, and trial metrics. It defines trial outcomes as binary labels derived from endpoints, phase advancement, and regulatory milestones, generated via a weakly supervised framework that combines multiple labeling signals with manual curation for recent trials (2020–2024). CTO achieves strong agreement with human labels, with F1 scores of approximately 0.94 for Phase III, 0.91 across all phases, and a manually curated benchmark yielding 0.971 on challenging cases; models trained on CTO labels outperform those trained on the fixed TOP benchmark, highlighting distribution shifts in recent trials. The authors publicly release the CTO knowledge base and annotations to support ongoing research and data-driven improvements in drug development.

Abstract

Background The cost of drug discovery and development is substantial, with clinical trial outcomes playing a critical role in regulatory approval and patient care. However, access to large-scale, high-quality clinical trial outcome data remains limited, hindering advancements in predictive modeling and evidence-based decision-making. Methods We present the Clinical Trial Outcome (CTO) benchmark, a fully reproducible, large-scale repository encompassing approximately 125,000 drug and biologics trials. CTO integrates large language model (LLM) interpretations of publications, trial phase progression tracking, sentiment analysis from news sources, stock price movements of trial sponsors, and additional trial-related metrics. Furthermore, we manually annotated a dataset of clinical trials conducted between 2020 and 2024 to enhance the quality and reliability of outcome labels. Results The trial outcome labels in the CTO benchmark agree strongly with expert annotations, achieving an F1 score of 94 for Phase 3 trials and 91 across all phases. Additionally, benchmarking standard machine learning models on our manually annotated dataset revealed distribution shifts in recent trials, underscoring the necessity of continuously updated labeling approaches. Conclusions By analyzing CTO's performance on recent clinical trials, we demonstrate the ongoing need for high-quality, up-to-date trial outcome labels. We publicly release the CTO knowledge base and annotated labels at https://chufangao.github.io/CTOD, with regular updates to support research on clinical trial outcomes and inform data-driven improvements in drug development.

Figures (9)

• Figure 1: Overview of our CTO benchmark pipeline. (A) Trial selection criteria used to curate a targeted dataset from the CTTI database, narrowing from an initial 513.4K trials to 125.8K trials in CTO to a final manually curated set of 11K trials. (B) Trial knowledge base creation pipeline, including extraction of linked PubMed abstracts, recent abstract searches, trial metrics, news articles, and stock prices. (C) Clinical trial outcome labeling process, integrating the CTO automated labeling framework assisted manual labeling to generate curated gold standard labels. (D) The manual curation process, our rule-based annotation criteria, followed by manual checking on challenging clinical trials.
• Figure 2: Overview of our manually curated trial outcome benchmark. (A) Bar plot illustrating the top 10 conditions in the benchmark, categorized by MeSH ancestor terms, with associated label counts. (B) Distribution of trial phases and their respective outcome labels. (C) Annual distribution of completed trials in the set from 2020-2024 and their label counts.
• Figure 3: (A) Year-by-year comparison of trial outcome prediction performance for baseline models trained on the TOP dataset, the constantly updated CTO dataset, and our manually curated set, evaluated on trials completed from 2021 to 2024. The TOP dataset is fully utilized for training, whereas for the CTO and manually curated sets, only trials completed before each test year are used for training. (B) Top 10 Condition-wise performance of baselines on manual labels, with models trained on data $<$ 2022 and tested on $\geq$ 2022 using TOP, CTO, and manually curated set. (C) Similar to B, Condition-wise performance on the next 10 (Top 11-20 conditions)
• Figure 4: Overview of the CTO dataset and labeling functions. (A)Trial distribution by completed year in CTO, which shows a steady increase over the years. (B) Frequency of top 30 conditions in the CTO dataset, categorized by MeSH ancestor terms. (C) Heatmap shows the agreement between the important labeling functions in CTO. (D) Distribution of trial phases and their counts in CTO.
• Figure 5: (A) provides the overview of the CTO automated labeling framework. CTO integrates various data sources to generate labels for predicting clinical trial outcomes. Data sources include (1) News articles, from which trial embeddings and sentiment scores are derived; (2) Publicly available stock prices, used to compute price changes before and after trials; (3) PubMed abstracts, where LLMs are prompted to predict the trial outcome; and (4) trial phase linkage. Additional sources of outcomes include trial metrics such as the number of patients, adverse events, etc. (B) Overview of trial linkage algorithm. (B1) Illustrates the phase connection map, covering all the phase categories present in the dataset. (B2) Linking trials across phases based on completion dates and intervention types, disease, and other trial data. We 1) constraint the search space, then 2) retrieve top-K most similar past trials using embeddings similarity. Finally, 3) a cross-encoder re-ranking strategy predicts linkages by scoring likely pairs. (B3) Matching FDA-approved drugs from FDA orange book to Phase 3 and Phase 2 $\&$ 3 drug trials.