Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining

TL;DR

ALTER tackles the challenge of learning from heterogeneous pathology data by enabling any-to-any tri-modal pretraining across whole-slide images, genomics, and pathology reports. It employs a modular modality encoder, a two-stage fusion with shared and modality-specific components, and three hierarchical pretraining tasks (MLM, CLIP, triplet) to align and generalize across missing modalities. Across 10 public datasets and four downstream tasks (survival, subtyping, mutation, report generation), ALTER demonstrates superior or competitive performance and robust generalization, including strong zero-shot behavior evidenced by attention heatmaps. This framework advances practical deployment in computational pathology by supporting flexible modality combinations and reducing data pairing requirements while preserving predictive power.

Abstract

Recent advances in computational pathology and artificial intelligence have significantly enhanced the utilization of gigapixel whole-slide images and and additional modalities (e.g., genomics) for pathological diagnosis. Although deep learning has demonstrated strong potential in pathology, several key challenges persist: (1) fusing heterogeneous data types requires sophisticated strategies beyond simple concatenation due to high computational costs; (2) common scenarios of missing modalities necessitate flexible strategies that allow the model to learn robustly in the absence of certain modalities; (3) the downstream tasks in CPath are diverse, ranging from unimodal to multimodal, cnecessitating a unified model capable of handling all modalities. To address these challenges, we propose ALTER, an any-to-any tri-modal pretraining framework that integrates WSIs, genomics, and pathology reports. The term "any" emphasizes ALTER's modality-adaptive design, enabling flexible pretraining with any subset of modalities, and its capacity to learn robust, cross-modal representations beyond WSI-centric approaches. We evaluate ALTER across extensive clinical tasks including survival prediction, cancer subtyping, gene mutation prediction, and report generation, achieving superior or comparable performance to state-of-the-art baselines.

Figures (5)

• Figure 1: Overview of our pretraining framework, ALTER. (a) ALTER processes each modality using modality-specific encoders, followed by a universal sequence Transformer and task-specific projection heads for downstream prediction. (b) The three-tiered constraints of ALTER, which can enable model to align multimodal inputs without requiring full modality pairing. (c) ALTER can be applied to any downstream task by integrating task-specific projection heads for fine-tuning.
• Figure 2: Universal sequence Transformer architecture of ALTER.
• Figure 3: ALTER's overall performance across all datasets. The base model denotes commonly used task-specific benchmarks (marked with † in § \ref{['baseline']}), while the suboptimal model refers to the model with the second-best overall performance for the specific task type.
• Figure 4: Analysis of the performance of ALTER. For cancer subtyping, a unimodal task, we freeze the fusion blocks to prevent multimodal information from being corrupted; for survival prediction as a multimodal task, we unfreeze them, where PT stands for pretraining, and FL stands for fusion layer.
• Figure 5: Illustration of model’s inspection capabilities on a patient of the LUAD study. The red dashed box highlights the high-risk regions that the trained multimodal model focuses on.