MultiST: A Cross-Attention-Based Multimodal Model for Spatial Transcriptomic

TL;DR

MultiST introduces a cross-attention–based multimodal framework that jointly models spatial gene expression and histology to improve spatial domain delineation, pseudotime reconstruction, and cell–cell interaction analysis in spatial transcriptomics. The method fuses a graph-based gene encoder with a color-normalized CLIP-ViT image encoder through bidirectional cross-attention, aided by masked expression learning, GAN-based latent refinement, and Fisher-MMD regularization to achieve stable, biologically coherent embeddings. Through three training stages, MultiST delivers more coherent domain boundaries, robust trajectories, and biologically interpretable signaling networks across 13 Visium datasets from human brain and breast cancer, outperforming existing baselines. The work demonstrates that integrating morphologic context with molecular profiles yields actionable insights for tissue organization, developmental trajectories, and tumor microenvironment interactions, with broad implications for spatial biology and translational oncology.

Abstract

Spatial transcriptomics (ST) enables transcriptome-wide profiling while preserving the spatial context of tissues, offering unprecedented opportunities to study tissue organization and cell-cell interactions in situ. Despite recent advances, existing methods often lack effective integration of histological morphology with molecular profiles, relying on shallow fusion strategies or omitting tissue images altogether, which limits their ability to resolve ambiguous spatial domain boundaries. To address this challenge, we propose MultiST, a unified multimodal framework that jointly models spatial topology, gene expression, and tissue morphology through cross-attention-based fusion. MultiST employs graph-based gene encoders with adversarial alignment to learn robust spatial representations, while integrating color-normalized histological features to capture molecular-morphological dependencies and refine domain boundaries. We evaluated the proposed method on 13 diverse ST datasets spanning two organs, including human brain cortex and breast cancer tissue. MultiST yields spatial domains with clearer and more coherent boundaries than existing methods, leading to more stable pseudotime trajectories and more biologically interpretable cell-cell interaction patterns. The MultiST framework and source code are available at https://github.com/LabJunBMI/MultiST.git.

Figures (6)

• Figure 1: The MultiST framework for spatially guided multimodal integration and biological discovery. Top: MultiST takes spatial transcriptomics data as input, including H&E-stained tissue images and matched gene expression matrices with spatial barcodes. Middle: The Spatially Guided Multimodal Integration module models molecular and histological modalities via two parallel encoders. Latent representations are fused through cross-attention, with label diffusion enforcing spatial coherence. Bottom: The refined multimodal representations support downstream analyses, including spatial domain identification, pseudotime trajectory inference, and cell--cell interaction (CCI).
• Figure 2: Graph-based spatial gene representation learning framework. The encoder integrates expression and spatial features into latent representations, with Fisher–MMD adversarial alignment and refined with deep embedded clustering.
• Figure 3: Color-Normalized Image Feature Extraction pipeline. Coordinate-aligned patches are encoded with a pretrained CLIP–ViT model, followed by KNN smoothing to enhance local consistency of the embeddings ($\tilde{V}$).
• Figure 4: MultiST consistently outperforms SOTA methods in clustering accuracy across 13 datasets.
• Figure 5: Biological validation of MultiST-based spatial clustering in breast cancer tissue sections. (A) Ground truth, H&E histology, and MultiST-predicted spatial domains. In the H&E panel, black outlines indicate ground-truth tissue regions, while red numbers mark the dominant MultiST clusters. (B) Representative marker genes for selected clusters. (C) Spatial pseudotime maps inferred by Monocle3. (D) KEGG enrichment analysis of MultiST clusters.