Multimodal Deep Learning for Prediction of Progression-Free Survival in Patients with Neuroendocrine Tumors Undergoing 177Lu-based Peptide Receptor Radionuclide Therapy

TL;DR

This study addresses predicting progression-free survival (PFS) in neuroendocrine tumor patients treated with PRRT by developing a multimodal deep learning framework that fuses pretherapy SR-PET, CT imaging, and laboratory biomarkers. Seven models compare unimodal versus multimodal approaches, with the best configuration—PET-CT-Lab fusion using a pretrained CT branch—achieving AUROC $0.72 \pm 0.01$ and AUPRC $0.80 \pm 0.01$, and enabling risk stratification via Kaplan–Meier analysis. The results show that imaging modalities alone are insufficient, but their integration with laboratory data yields robust predictive performance and interpretable signals (e.g., importance of CgA and GGT; gradient-based localization to tumorous regions). These findings support risk-adapted follow-up strategies and warrant external validation in larger, multicenter cohorts to enable clinical translation.

Abstract

Peptide receptor radionuclide therapy (PRRT) is an established treatment for metastatic neuroendocrine tumors (NETs), yet long-term disease control occurs only in a subset of patients. Predicting progression-free survival (PFS) could support individualized treatment planning. This study evaluates laboratory, imaging, and multimodal deep learning models for PFS prediction in PRRT-treated patients. In this retrospective, single-center study 116 patients with metastatic NETs undergoing 177Lu-DOTATOC were included. Clinical characteristics, laboratory values, and pretherapeutic somatostatin receptor positron emission tomography/computed tomographies (SR-PET/CT) were collected. Seven models were trained to classify low- vs. high-PFS groups, including unimodal (laboratory, SR-PET, or CT) and multimodal fusion approaches. Explainability was evaluated by feature importance analysis and gradient maps. Forty-two patients (36%) had short PFS (< 1 year), 74 patients long PFS (>1 year). Groups were similar in most characteristics, except for higher baseline chromogranin A (p = 0.003), elevated gamma-GT (p = 0.002), and fewer PRRT cycles (p < 0.001) in short-PFS patients. The Random Forest model trained only on laboratory biomarkers reached an AUROC of 0.59 +- 0.02. Unimodal three-dimensional convolutional neural networks using SR-PET or CT performed worse (AUROC 0.42 +- 0.03 and 0.54 +- 0.01, respectively). A multimodal fusion model laboratory values, SR-PET, and CT -augmented with a pretrained CT branch - achieved the best results (AUROC 0.72 +- 0.01, AUPRC 0.80 +- 0.01). Multimodal deep learning combining SR-PET, CT, and laboratory biomarkers outperformed unimodal approaches for PFS prediction after PRRT. Upon external validation, such models may support risk-adapted follow-up strategies.

Figures (9)

• Figure 1: (Top) Overview of the proposed deep learning pipeline for PRRT response prediction. The model integrates 3D PET and 3D CT scans processed through separate 3D convolutional networks, along with laboratory biomarkers, via a concatenation-based fusion layer. The fused features are passed through fully connected layers to generate the prediction output. Interpretability is provided through backward analyses, including gradient maps, feature importance, and UMAP-based feature space analysis. Solid arrows represent forward pass data flow, dashed arrows backward pass dervied post hoc outputs. (Bottom) Overview of input modality combinations evaluated in our experiments.
• Figure 2: Kaplan-Meier curve for progression-free survival (PFS) in the total study cohort (n = 116) of patients with neuroendocrine tumors treated with [177Lu]Lu-DOTATOC PRRT. Vertical dashed red line indicates our split into high and low therapy response. No censored patients are included, and all patients eventually had progress.
• Figure 3: Barplot comparison of AUROC and AUPRC across models.
• Figure 4: Comparison of predictive performance between the Random Forest baseline (laboratory values only) model and the PET CT Fusion model. The plot is showing the example of a single representative cv fold. Left: ROC curves showing True Positive Rate versus False Positive Rate; the dashed gray line represents a random baseline. Right: Precision-Recall curves illustrating the trade-off between precision and recall; the dashed gray line indicates the all-positive baseline. The PET CT Fusion model consistently outperforms the Random Forest baseline, as reflected in higher AUROC and AUPRC values. For cross-validation metrics, refer to Table \ref{['tab:model_performance']}.
• Figure 5: Kaplan-Meier Curve of study cohort, stratified by our model prediction output probabilities $\hat{y}$ (low PFS: $\hat{y} < 0.5$, high PFS: $\hat{y} >= 0.5$). Note that as we did not include censored patients, and all patients in our cohort eventually had progression, the y axis represents the proportion of progression patients at a given time. Log-Rank Test: $p = 0.0001$.