LaCoGSEA: Unsupervised deep learning for pathway analysis via latent correlation

TL;DR

LaCoGSEA tackles the limitation of traditional GSEA, which depends on supervised labels, by introducing an unsupervised pre-ranking mechanism. It leverages a deep autoencoder to capture non-linear transcriptomic structure and derives dense gene rankings from global gene–latent correlations, enabling seamless application of GSEA without phenotypic labels. Across diverse cancer and disease datasets, LaCoGSEA improves pathway recovery, enables robust subtype stratification, and demonstrates cross-dataset consistency, outperforming linear methods and gradient-based XAI approaches. The framework provides a practical, interpretable tool for uncovering pathway-level biology in heterogeneous and small-sample transcriptomic studies.

Abstract

Motivation: Pathway enrichment analysis is widely used to interpret gene expression data. Standard approaches, such as GSEA, rely on predefined phenotypic labels and pairwise comparisons, which limits their applicability in unsupervised settings. Existing unsupervised extensions, including single-sample methods, provide pathway-level summaries but primarily capture linear relationships and do not explicitly model gene-pathway associations. More recently, deep learning models have been explored to capture non-linear transcriptomic structure. However, their interpretation has typically relied on generic explainable AI (XAI) techniques designed for feature-level attribution. As these methods are not designed for pathway-level interpretation in unsupervised transcriptomic analyses, their effectiveness in this setting remains limited. Results: To bridge this gap, we introduce LaCoGSEA (Latent Correlation GSEA), an unsupervised framework that integrates deep representation learning with robust pathway statistics. LaCoGSEA employs an autoencoder to capture non-linear manifolds and proposes a global gene-latent correlation metric as a proxy for differential expression, generating dense gene rankings without prior labels. We demonstrate that LaCoGSEA offers three key advantages: (i) it achieves improved clustering performance in distinguishing cancer subtypes compared to existing unsupervised baselines; (ii) it recovers a broader range of biologically meaningful pathways at higher ranks compared with linear dimensionality reduction and gradient-based XAI methods; and (iii) it maintains high robustness and consistency across varying experimental protocols and dataset sizes. Overall, LaCoGSEA provides state-of-the-art performance in unsupervised pathway enrichment analysis. Availability and implementation: https://github.com/willyzzz/LaCoGSEA

Figures (5)

• Figure 1: Schematic overview of the LaCoGSEA framework. (a) Bulk transcriptomic data are first compressed into low-dimensional latent embeddings using a deep autoencoder. Gene–dimension correlations are then computed between the original gene expression and each latent dimension, producing a gene–dimension correlation map. These correlations are used to generate pre-ranked gene lists for standard Gene Set Enrichment Analysis (GSEA). (b) Application to the SCAN-B dataset with latent dimension $D=4$. For each latent dimension, the top 5 enriched KEGG pathways are reported, which shows with different biology mechanisms. (c) The overall sample-level pathway activity matrix for SCAN-B ($D=4$) is generated and visualized using t-SNE, illustrating coherent separation of samples based on pathway programs.
• Figure 2: Saturation analysis reveals superior pathway detection capacity of the LaCoGSEA compared to PCA. (a–f) Comparison of the number of significant pathways detected by AE (blue) and PCA (purple) across increasing latent space dimensions ($D \in {1, 2, 4, 8, \dots, 128}$). Analysis was performed on SCAN-B (a, b, c) and METABRIC (d, e, f) cohorts using three gene set collections: GO Biological Processes, KEGG, and C6 Oncogenic Signatures. Significance was defined using a strict Bonferroni-corrected threshold (FDR $< 0.05/D$).
• Figure 3: Latent pathway activity inference enhances breast cancer subtype clustering. (a–d) t-SNE visualizations of pathway activity scores generated by (a) LaCoGSEA, (b) PCA, (c) GSVA, and (d) ssGSEA. Points are colored by PAM50 subtypes (Basal, LumA, LumB, HER2, Normal). (e) Bar chart showing the Adjusted Rand Index (ARI) for clustering performance on the SCAN-B cohort using KEGG pathways. The LaCoGSEA pathway activity matrix achieves the highest concordance with PAM50 clinical subtypes (ARI = 0.372), significantly outperforming PCA (0.240), GSVA (0.126), and ssGSEA (0.185). Error bars represent standard deviation across bootstrap iterations.
• Figure 4: LaCoGSEA identifies distinct metabolic and proliferative drivers across breast and lung cancer subtypes.(a) Differential pathway enrichment analysis between Basal-like and Luminal A subtypes in the SCAN-B cohort. The model captures a fundamental metabolic switch: Basal tumors (red) are dominated by proliferation and DNA repair mechanisms (e.g., P53 signaling, Cell cycle), whereas LumA tumors (blue) are characterized by oxidative and lipid metabolism (e.g., Peroxisome, Fatty acid metabolism). (b--c) Box plots of inferred pathway activity scores validating the enrichment results. (b)P53 signaling is significantly elevated in Basal samples ($P < 0.0001$), while (c)Peroxisome activity is significantly higher in Luminal subtypes, consistent with their dependency on fatty acid oxidation. (d) Differential pathway signatures distinguishing Lung Squamous Cell Carcinoma (LUSC) from Lung Adenocarcinoma (LUAD) in the TCGA dataset. LaCoGSEA identifies Steroid hormone biosynthesis as a primary driver for LUSC. (e--f) Validation of subtype-specific activity scores. (e)Steroid hormone biosynthesis shows robust upregulation in LUSC, while (f)Antigen processing and presentation is significantly enriched in LUAD ($P < 0.0001$, Wilcoxon rank-sum test). Error bars represent the interquartile range.
• Figure 5: LaCoGSEA with global correlation metrics outperforms linear and gradient-based methods in target pathway prioritization. (a--b) Heatmaps displaying the ranking of target pathways across different computational methods for (a) GSE126848 (Liver Disease) and (b) GSE48350 (Alzheimer's Disease). Rows represent known target pathways, and columns represent methods. The color intensity represents the rank (darker blue indicates a lower/better rank). LaCoGSEA consistently assigns the lowest ranks to validated mechanisms compared to PCA baselines (PCA_Corr, PCA_Weights) and deep learning attribution methods (AE_DeepLIFT, AE_SHAP). (c) Summary of method performance across five independent benchmarking datasets. The dot plot shows the Mean Target Pathway Rank (lower is better) for each method. If a target pathway was not detected, a penalty rank of 100 was assigned. The vertical black line indicates the mean rank across all datasets. LaCoGSEA achieves the best overall performance (Mean Rank $\approx 17.5$), significantly outperforming standard DE and interpretability-based AE variants.