Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations

TL;DR

This study investigates whether vision pathology foundation models encode continuous disease progression in their embeddings, rather than only discrete class labels. By applying diffusion pseudotime to patch representations from six models across four cancer progression cohorts, the authors show that trajectory-like structure emerges as a low-dimensional manifold, with trajectory fidelity measured by Kendall's $\tau$ correlating with few-shot generalization (mean $\rho$ ≈ 0.92 across held-out tasks). Vision-only pathology models consistently achieve the highest trajectory fidelity, while a natural image baseline performs weaker or variably, indicating domain-specific pretraining is key. The results further reveal biologically plausible shifts in cell-type composition along pseudotime in colorectal progression, illustrating that learned representations capture continuity beyond standard classification, and offering a general framework for analyzing continuous processes in static image data.

Abstract

Vision foundation models trained on discretely sampled images achieve strong performance on classification benchmarks, yet whether their representations encode the continuous processes underlying their training data remains unclear. This question is especially pertinent in computational pathology, where we posit that models whose latent representations implicitly capture continuous disease progression may better reflect underlying biology, support more robust generalization, and enable quantitative analyses of features associated with disease transitions. Using diffusion pseudotime, a method developed to infer developmental trajectories from single-cell transcriptomics, we probe whether foundation models organize disease states along coherent progression directions in representation space. Across four cancer progressions and six models, we find that all pathology-specific models recover trajectory orderings significantly exceeding null baselines, with vision-only models achieving the highest fidelities $(τ> 0.78$ on CRC-Serrated). Model rankings by trajectory fidelity on reference diseases strongly predict few-shot classification performance on held-out diseases ($ρ= 0.92$), and exploratory analysis shows cell-type composition varies smoothly along inferred trajectories in patterns consistent with known stromal remodeling. Together, these results demonstrate that vision foundation models can implicitly learn to represent continuous processes from independent static observations, and that trajectory fidelity provides a complementary measure of representation quality beyond downstream performance. While demonstrated in pathology, this framework could be applied to other domains where continuous processes are observed through static snapshots.

Figures (21)

• Figure 1: Overview of trajectory fidelity evaluation. H&E patches representing disease states (here: skin squamous cell carcinoma progression from epidermis through actinic keratosis and carcinoma in situ to invasive SCC) are embedded by a foundation model. We apply diffusion pseudotime analysis to the embedding space, visualized here via diffusion components (DC1, DC2). Left: patches colored by ground-truth class show spatial organization reflecting disease progression. Right: the same embedding colored by inferred pseudotime reveals a continuous gradient. Trajectory fidelity ($\tau$) quantifies agreement between pseudotime ordering and ground-truth ordinal class labels using Kendall's rank correlation; $\tau = 0.72$ indicates strong preservation of biological progression structure.
• Figure 2: Trajectory fidelity across foundation models and disease progressions. Kendall's $\tau$ between diffusion pseudotime and ground-truth class labels for six foundation models across four cancer progressions. All models significantly exceed the null baseline (gray ×, label-shuffled controls). Vision-only pathology models (UNI-2, Virchow-2, Prov-GigaPath) achieve the highest and most consistent fidelity, while the natural image baseline (DINOv2) shows weaker and more variable recovery. Exact values reported in Appendix \ref{['app:traj_fidel_results']}.
• Figure 3: Trajectory fidelity reflects biological ordering, not class structure. Stars: mean $\tau$ for true progression order across four cohorts (95% CI). Boxplots: null distributions from permuted class label orderings, pooled across cohorts. All models exceed the null median; vision-only CPath models show clearest separation.
• Figure 4: Trajectory fidelity emerges progressively across network depth.Top: Trajectory fidelity ($\tau$) using [CLS] token embeddings, averaged across progressions. Vision-only pathology models (UNI-2, Virchow-2, Prov-GigaPath) and CONCH show steady improvement through 87.5--100% depth, while MuSK exhibits a mid-network dip before recovery. Bottom: Average intrinsic dimensionality of raw embeddings (blue) expands across layers ($\sim$12 to $\sim$18) while diffusion manifold dimensionality (red) remains stable ($\approx$4--5). Shaded regions show 95% CI across models and progressions.
• Figure 5: Model rankings by trajectory fidelity transfer to held-out diseases. Cross-task generalization using a leave-one-out protocol. For each target disease (panel), we ranked models by mean $\tau$ on the remaining cohorts (x-axis) and by 5-shot F1 on the held-out target (y-axis). Diagonal indicates perfect rank preservation. Model rankings transfer reliably across tissue types.