hSNMF: Hybrid Spatially Regularized NMF for Image-Derived Spatial Transcriptomics

TL;DR

This work tackles the challenge of learning interpretable representations from high-dimensional spatial transcriptomics data by extending nonnegative matrix factorization with spatial regularization. It introduces Spatial NMF (SNMF) as a lightweight, spatially smoothed two-stage embedding and Hybrid Spatial NMF (hSNMF), which combines spatial proximity and transcriptomic similarity via a contact–radius hybrid graph and Leiden clustering. On Xenium-derived cholangiocarcinoma data, SNMF and especially hSNMF yield tighter spatial clusters (low CHAOS) and strong spatial autocorrelation (high Moran's I), coupled with improved biological coherence (CMC and enrichment). The methods are scalable, interpretable, and provide a practical framework for uncovering spatially organized cell states in high-dimensional ST data, with code available at GitHub.

Abstract

High-resolution spatial transcriptomics platforms, such as Xenium, generate single-cell images that capture both molecular and spatial context, but their extremely high dimensionality poses major challenges for representation learning and clustering. In this study, we analyze data from the Xenium platform, which captures high-resolution images of tumor microarray (TMA) tissues and converts them into cell-by-gene matrices suitable for computational analysis. We benchmark and extend nonnegative matrix factorization (NMF) for spatial transcriptomics by introducing two spatially regularized variants. First, we propose Spatial NMF (SNMF), a lightweight baseline that enforces local spatial smoothness by diffusing each cell's NMF factor vector over its spatial neighborhood. Second, we introduce Hybrid Spatial NMF (hSNMF), which performs spatially regularized NMF followed by Leiden clustering on a hybrid adjacency that integrates spatial proximity (via a contact-radius graph) and transcriptomic similarity through a tunable mixing parameter alpha. Evaluated on a cholangiocarcinoma dataset, SNMF and hSNMF achieve markedly improved spatial compactness (CHAOS < 0.004, Moran's I > 0.96), greater cluster separability (Silhouette > 0.12, DBI < 1.8), and higher biological coherence (CMC and enrichment) compared to other spatial baselines. Availability and implementation: https://github.com/ishtyaqmahmud/hSNMF

Figures (1)

• Figure 1: UMAP projections of cell clusters obtained after applying Leiden clustering on the reduces cellxgene vectors.