SENCA-st: Integrating Spatial Transcriptomics and Histopathology with Cross Attention Shared Encoder for Region Identification in Cancer Pathology

TL;DR

The paper tackles the problem of identifying functionally distinct tumor regions by integrating spatial transcriptomics with histopathology, addressing the noise in transcriptomic data and the risk of structure-dominated fusion. It introduces SENCA-st, a dual-branch architecture with a neighborhood cross-attention shared encoder and hierarchical learning to fuse modalities while preserving local functional signals. The method achieves state-of-the-art region segmentation performance on breast cancer (HER2ST) and squamous cell carcinoma datasets, with quantitative ARI gains and qualitative validation, supported by statistical significance. The approach enables self-supervised, zero-shot region discovery and offers a pathway toward biomarker identification and improved understanding of tumor heterogeneity and micro-environment interactions in cancer pathology.

Abstract

Spatial transcriptomics is an emerging field that enables the identification of functional regions based on the spatial distribution of gene expression. Integrating this functional information present in transcriptomic data with structural data from histopathology images is an active research area with applications in identifying tumor substructures associated with cancer drug resistance. Current histopathology-spatial-transcriptomic region segmentation methods suffer due to either making spatial transcriptomics prominent by using histopathology features just to assist processing spatial transcriptomics data or using vanilla contrastive learning that make histopathology images prominent due to only promoting common features losing functional information. In both extremes, the model gets either lost in the noise of spatial transcriptomics or overly smoothed, losing essential information. Thus, we propose our novel architecture SENCA-st (Shared Encoder with Neighborhood Cross Attention) that preserves the features of both modalities. More importantly, it emphasizes regions that are structurally similar in histopathology but functionally different on spatial transcriptomics using cross-attention. We demonstrate the superior performance of our model that surpasses state-of-the-art methods in detecting tumor heterogeneity and tumor micro-environment regions, a clinically crucial aspect.

Figures (5)

• Figure 1: SENCA-st system architecture: we leverage both spatial transcriptomics and histopathology data processed through a graph transformer and a ResNet encoder to produce embeddings through neighborhood cross-attention shared encoder which leads to better segmentation.
• Figure 2: Qualitative comparison for H1 sample in HER2ST: SENCA-st (our) clustering closely matches with the ground truth. For this particular sample ARI is high at 0.518.
• Figure 3: Spatial Clusters of sample p2 of Squamous Cell Carcinoma with marker genes.
• Figure 4: Ablation Study - Qualitative effect visualization on H1 sample of HER2ST dataset. We investigated effect of components of the system by removing them and conducting ablation studies.
• Figure 5: Clustering of shared embeddings generated by SENCA-st of C1 sample of HER2ST Dataset