AI-driven 3D Spatial Transcriptomics

TL;DR

VORTEX addresses the bottleneck of scalable 3D spatial transcriptomics by predicting volumetric 3D ST from 3D tissue morphology and limited 2D ST data. It is pretrained on diverse 3D morphology–transcriptomic pairs and fine-tuned on minimal VOI 2D ST data to capture both general morphomolecular patterns and sample-specific links. The model combines a 2D image encoder (CONCH), a 3D image encoder (ViT-based), and a transcriptomics encoder (scGPT), trained with contrastive and ST reconstruction losses, enabling cross-modal alignment and 3D ST prediction across large tissue volumes. In prostate cancer datasets, 3D+VOI fine-tuning yields superior predictions, demonstrates scalability to large volumes, and enables cross-modal retrieval for biomarker exploration, offering a cost-effective, minimally destructive route to volumetric molecular insights with potential to accelerate biomarker discovery and morphomolecular understanding.

Abstract

A comprehensive three-dimensional (3D) map of tissue architecture and gene expression is crucial for illuminating the complexity and heterogeneity of tissues across diverse biomedical applications. However, most spatial transcriptomics (ST) approaches remain limited to two-dimensional (2D) sections of tissue. Although current 3D ST methods hold promise, they typically require extensive tissue sectioning, are complex, are not compatible with non-destructive 3D tissue imaging technologies, and often lack scalability. Here, we present VOlumetrically Resolved Transcriptomics EXpression (VORTEX), an AI framework that leverages 3D tissue morphology and minimal 2D ST to predict volumetric 3D ST. By pretraining on diverse 3D morphology-transcriptomic pairs from heterogeneous tissue samples and then fine-tuning on minimal 2D ST data from a specific volume of interest, VORTEX learns both generic tissue-related and sample-specific morphological correlates of gene expression. This approach enables dense, high-throughput, and fast 3D ST, scaling seamlessly to large tissue volumes far beyond the reach of existing 3D ST techniques. By offering a cost-effective and minimally destructive route to obtaining volumetric molecular insights, we anticipate that VORTEX will accelerate biomarker discovery and our understanding of morphomolecular associations and cell states in complex tissues. Interactive 3D ST volumes can be viewed at https://vortex-demo.github.io/

Figures (14)

• Figure 1: Overview of $\textsc{VORTEX}$. (a) Workflow of 3D ST prediction with $\textsc{VORTEX}$ on a test volume (or volume of interest, VOI). $\textsc{VORTEX}$ provides efficient whole volume 3D ST prediction for gene sets of interest based on the 3D tissue images and ST measurements. A 3D tissue image is obtained with a non-destructive 3D imaging modality (microCT chosen as an illustrative example). ST is performed on a few 2D tissue sections from the same tissue volume (Visium chosen as an illustrative example). $\textsc{VORTEX}$ is trained over two stages. It is first pretrained on a disease-specific cohort of 2D (or 3D) tissue images and 2D ST data pairs. It is further fine-tuned on data pairs of 2D (or 3D) tissue images and 2D ST acquired from the VOI. $\textsc{VORTEX}$ can also be extended to 2.5D tissue images comprised of serial tissue sections. (b) Illustration of $\textsc{VORTEX}$ architecture. All deep learning components of $\textsc{VORTEX}$ are trained with a combination of ST reconstruction and cross-modal alignment loss. The green arrows indicate the prediction workflow of $\textsc{VORTEX}$ once trained. (c) Applications for $\textsc{VORTEX}$ on efficient large 3D ST prediction, joint morphology and ST analysis, and 3D morphology query. ST: Spatial transcriptomics. Morph. Seg.: Morphological segmentation.
• Figure 1: $\textsc{VORTEX}$ ST prediction analysis on additional gene sets. In addition to the analysis for 250 highly-expressed genes (HEG) in Figure \ref{['fig:prostate']}, we analyze $\textsc{VORTEX}$ for gene sets with (a) 1,000 HEG and (b) 250 highly variable genes (HVG) over three different scenarios. Error bars indicate one standard deviation from the mean, over ten sections across five patients. In addition to the analysis for 250 highly-expressed genes (HEG) in Extended Figure \ref{['fig:breast_crc']}, we analyze $\textsc{VORTEX}$ for gene sets with 1,000 HEG over two different scenarios for (c) the breast cancer cohort and (d) the colorectal cancer cohort. Statistical significance was assessed with the Wilcoxon signed-rank test. $^{\ast\ast}p\leq 0.01$, $^{\ast\ast\ast}p\leq 0.001$, $^{\ast\ast\ast\ast}p\leq 0.0001$. PCC: Pearson Correlation Coefficient. SSIM: Structural Similarity Index Measure.
• Figure 2: $\textsc{VORTEX}$ analysis on prostate cancer. (a) Cross-modal registration between the 3D microCT tissue image (4$\mu m$/voxel) and the H&E-stained tissue sections with 2D ST. Checkerboard visualization of co-registered microCT and H&E images. (b) Schematics for different training scenarios: 2D image and 2D ST pairs (2D), adding 3D image and 2D ST pairs (3D), and further adding 3D image and 2D ST pairs from the VOI (3D + VOI). (c) PCC and SSIM between the predicted and the measured expression for three gene sets for five patients: All Genes (264 genes), the top 50 highly predictive genes, and marker genes. Error bars indicate one standard deviation from the mean, over ten sections across five patients. (d) Spearman's $\rho$ between the variance of measured and predicted expressions across all genes. Each black dot represents a tissue section. (e) Difference in Moran's I and Geary's C metric between measured and predicted expressions aggregated across all genes in five patients. (f) 3D ST prediction heatmap for select genes, with 3D morphological segmentation masks. Additional examples can be found in Extended Data Figure \ref{['fig:ext_prostate_3D']}. (g) Cross-section visualizations of morphology, measured and predicted ST for AZGP1, and morphological segmentation masks. (h) ARI metrics across the depth of tissue volume. The ARI metric is measured between the segmentation mask and predicted spatial domains. Each black dot represents a tissue section. (i) The spatial domains identified by $\textsc{VORTEX}$ for the plane at $400 \mu m$. All scalebars are $1\,mm$. Statistical significance was assessed with the Wilcoxon signed-rank test. $^{\ast}p\leq 0.05$, $^{\ast\ast}p\leq 0.01$, $^{\ast\ast\ast}p\leq 0.001$. Whiskers extend to data points within 1.5$\times$ the interquartile range. VOI: Volume of interest. PCC: Pearson Correlation Coefficient. SSIM: Structural Similarity Index Measure. ARI: Adjusted Rand Index.
• Figure 2: $\textsc{VORTEX}$ ST prediction analysis on gene expression variance. The correlation Spearman's $\rho$ between the variance of $\textsc{VORTEX}$-predicted expression levels (orange) and the variance of measured ST expression levels (blue) across all Visium ST spots in each tissue section (refer to Online Methods in section ST spot filtering and expression normalization). Genes are ranked based on measured ST expression variance, from the smallest to the largest. The variance measures are shown across three different scenarios for three exemplar sections.
• Figure 3: $\textsc{VORTEX}$ on large prostate cancer tissue. (a) 3D ST prediction by $\textsc{VORTEX}$ on large prostate cancer tissue volume for EpCAM and ACTA2 genes, with the spatial domains identified by $\textsc{VORTEX}$. Cross-sections at 620 $\mu m$ and 1,120 $\mu m$ are also displayed. The red box indicates the ST capture area at the depth of 490 $\mu m$. The corresponding H&E tissue image based on which the ST capture area was selected is also shown. (b) Zoomed-in regions-of-interests from the tissue section at depth 1,120 $\mu m$. (c) The coronal and sagittal plane of the tissue volume and the corresponding prediction for EpCAM and ACTA2. Additional examples can be found in Extended Data Figure \ref{['fig:ext_prostate_3D_largeFOV']}. Scalebar is 1 mm.