CLaDMoP: Learning Transferrable Models from Successful Clinical Trials via LLMs

TL;DR

CLaDMoP tackles clinical trial outcome prediction under limited labeled data by combining a large language model branch for eligibility criteria with a lightweight drug-molecule branch, connected through multi-level fusion and grouping blocks. It introduces the Successful Clinical Trials (SCT) dataset and uses a pair-matching contrastive pretraining objective ($\mathcal{L}_{\mathrm{InfoNCE}}$) while freezing the LLM, followed by PEFT-based fine-tuning with LoRA to achieve strong performance on the Trial Outcome Prediction (TOP) benchmark. The approach yields up to $10.5\%$ PR-AUC and $3.6\%$ ROC-AUC gains over baselines, with notable improvements in Phase I/II and good generalization to new diseases. These results demonstrate that task-agnostic pretraining combined with efficient cross-branch fusion can produce robust, transferable representations for clinical trial outcome prediction and enable data-efficient adaptation to new trials.

Abstract

Many existing models for clinical trial outcome prediction are optimized using task-specific loss functions on trial phase-specific data. While this scheme may boost prediction for common diseases and drugs, it can hinder learning of generalizable representations, leading to more false positives/negatives. To address this limitation, we introduce CLaDMoP, a new pre-training approach for clinical trial outcome prediction, alongside the Successful Clinical Trials dataset(SCT), specifically designed for this task. CLaDMoP leverages a Large Language Model-to encode trials' eligibility criteria-linked to a lightweight Drug-Molecule branch through a novel multi-level fusion technique. To efficiently fuse long embeddings across levels, we incorporate a grouping block, drastically reducing computational overhead. CLaDMoP avoids reliance on task-specific objectives by pre-training on a "pair matching" proxy task. Compared to established zero-shot and few-shot baselines, our method significantly improves both PR-AUC and ROC-AUC, especially for phase I and phase II trials. We further evaluate and perform ablation on CLaDMoP after Parameter-Efficient Fine-Tuning, comparing it to state-of-the-art supervised baselines, including MEXA-CTP, on the Trial Outcome Prediction(TOP) benchmark. CLaDMoP achieves up to 10.5% improvement in PR-AUC and 3.6% in ROC-AUC, while attaining comparable F1 score to MEXA-CTP, highlighting its potential for clinical trial outcome prediction. Code and SCT dataset can be downloaded from https://github.com/murai-lab/CLaDMoP.

Figures (8)

• Figure 1: (a) Recent work has focused on using a lightweight encoder model and a prediction head trained with a task-specific loss due to limitations in the availability of labeled data. (b) In contrast, CLaDMoP follows a two-stage training paradigm. It incorporates a large language model (LLM branch) and a lightweight attention model (Disease-Molecule branch), sharing knowledge from the former to the latter. (b-left) Pre-training stage: uses pair matching as a proxy task and a task-agnostic loss for optimization. (b-right) Fine-tuning stage: evaluation performance is further improved by training a prediction head and LoRA layers for DM branch.
• Figure 2: (Left) CLaDMoP consists of two branches: the LLM branch and the DM branch. The prediction head is a three-layer residual network, which is trained during the PEFT stage. Further details are provided in Section \ref{['sec:model arc']}. (Right-top) The grouping block enables the model to fuse information from the LLM branch. Each grouping block consists of $G$ grouping layer(s), where the number of centroids is progressively reduced at each layer. The structure of the grouping layer is shown in (Right-bottom), while $S$ indicates the number of the self-attention layer in a grouping layer, with further details provided in Section \ref{['sec:grouping']} . Note that $u$ is a single token belonging to token sequence $U$.
• Figure 3: Few-shot learning results for clinical trial outcome prediction for phase I, II and III trials.
• Figure 4: Few-shot learning results for clinical trial outcome prediction for phase I, II and III trials.
• Figure 5: We show three different training strategies for model development, showing the dataflow during both pretraining and fine-tuning stages. The arrows represent the path of the data usage for each strategy: (i) Split-Split Training Strategy, where separate models are trained on different phases in SCT and then fine-tuned on the corresponding phases in TOP; (ii) Black Arrows: Full-Full training strategy, where the entire SCT is used for pre-training and entire TOP is used for fine-tuning; (iii) Red Arrows: Full-Split Training Strategy, where the entire SCT dataset is used for pretraining, while fine-tuning is performed on phase-specific TOP dataset.

Theorems & Definitions (6)

• definition 1
• definition 2
• definition 3
• definition 4
• definition 5
• definition 6