Multimodal Cancer Modeling in the Age of Foundation Model Embeddings

TL;DR

This work treats TCGA as a rich multimodal resource and proposes an embedding-centric framework that leverages zero-shot foundation-model embeddings across modalities to predict cancer survival with simple CoxPH models. By combining RNA-seq, histology, and pathology-report text (summarized by LLMs) through a late-fusion approach, the authors demonstrate additive gains over unimodal embeddings and establish a practical, modular pipeline that does not require training large neural nets. Key findings include substantial performance gains from multimodal fusion (best C-index ~0.795 when combined with demographics) and a strong positive impact of pathology-report summarization on prognostic accuracy, while hallucination corrections had minimal effect on risk stratification. The study highlights a scalable, privacy-conscious paradigm for leveraging foundation-model embeddings in cancer prognostics and lays groundwork for broader application to multi-omics and biomedical text data.

Abstract

The Cancer Genome Atlas (TCGA) has enabled novel discoveries and served as a large-scale reference dataset in cancer through its harmonized genomics, clinical, and imaging data. Numerous prior studies have developed bespoke deep learning models over TCGA for tasks such as cancer survival prediction. A modern paradigm in biomedical deep learning is the development of foundation models (FMs) to derive feature embeddings agnostic to a specific modeling task. Biomedical text especially has seen growing development of FMs. While TCGA contains free-text data as pathology reports, these have been historically underutilized. Here, we investigate the ability to train classical machine learning models over multimodal, zero-shot FM embeddings of cancer data. We demonstrate the ease and additive effect of multimodal fusion, outperforming unimodal models. Further, we show the benefit of including pathology report text and rigorously evaluate the effect of model-based text summarization and hallucination. Overall, we propose an embedding-centric approach to multimodal cancer modeling.

Figures (7)

• Figure 1: Multimodal cancer modeling is vastly simplified in the age of foundation models. Simple ensembles of small, classical models over zero-shot FM embeddings improve pan-cancer survival risk stratification. Embedding methods are modular, allowing for simple experimentation and orchestration.
• Figure 2: Eight thousand patient cases spanning 32 cancer types with survival data, pathology reports, diagnostic slides, and gene expression quantification.
• Figure 3: Summarization of pathology reports improves survival model risk stratification over unimodal text embeddings. Embeddings derived with BioMistral. Reports summarized with Llama-3.1-8B-Instruct. Averaged risk stratification from 5-fold cross-validation.
• Figure 4: Manual correction of summarized pathology report hallucinations does not impact survival model risk stratification. Embeddings derived with BioMistral. Reports summarized with Llama-3.1-8B-Instruct. Risk stratification from N=40 randomly sampled cases contained within a single test split while preserving observed mortality prevalence.
• Figure 5: Modality specificity of gene expression embedding model improves survival model risk stratification over unimodal gene expression embeddings. Bulk RNA-seq data embedded with either BulkRNABert or UCE, a single-cell RNA-seq model. Averaged risk stratification from 5-fold cross-validation.