Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rare Genomic Subtype Discovery from RNA-seq via Autoencoder Embeddings and Stability-Aware Clustering

Alaa Mezghiche

TL;DR

Rare transcriptomic subtypes in cancer can be obscured by dominant tissue-of-origin signals in pan-cancer RNA-seq data. The authors combine a compact autoencoder-based latent representation with stability-aware clustering to identify rare, reproducible subtypes, validating the approach with a pan-cancer negative control that shows tissue-driven clustering (Cramér’s $V = 0.887$) and then focusing on KIRC to detect a rare C0 subtype at $k = 5$ with prevalence $6.85\%$ and Jaccard $0.787$. A differential-expression analysis yields a coherent marker signature for C0, underscoring the biological relevance of the subtype. This workflow provides a practical framework for unsupervised discovery of rare genomic subtypes in RNA-seq data and can be extended to richer datasets like TCGA to link transcriptional states with clinical outcomes and therapies.

Abstract

Unsupervised learning on high-dimensional RNA-seq data can reveal molecular subtypes beyond standard labels. We combine an autoencoder-based representation with clustering and stability analysis to search for rare but reproducible genomic subtypes. On the UCI "Gene Expression Cancer RNA-Seq" dataset (801 samples, 20,531 genes; BRCA, COAD, KIRC, LUAD, PRAD), a pan-cancer analysis shows clusters aligning almost perfectly with tissue of origin (Cramer's V = 0.887), serving as a negative control. We therefore reframe the problem within KIRC (n = 146): we select the top 2,000 highly variable genes, standardize them, train a feed-forward autoencoder (128-dimensional latent space), and run k-means for k = 2-10. While global indices favor small k, scanning k with a pre-specified discovery rule (rare < 10 percent and stable with Jaccard >= 0.60 across 20 seeds after Hungarian alignment) yields a simple solution at k = 5 (silhouette = 0.129, DBI = 2.045) with a rare cluster C0 (6.85 percent of patients) that is highly stable (Jaccard = 0.787). Cluster-vs-rest differential expression (Welch's t-test, Benjamini-Hochberg FDR) identifies coherent markers. Overall, pan-cancer clustering is dominated by tissue of origin, whereas a stability-aware within-cancer approach reveals a rare, reproducible KIRC subtype.

Rare Genomic Subtype Discovery from RNA-seq via Autoencoder Embeddings and Stability-Aware Clustering

TL;DR

Rare transcriptomic subtypes in cancer can be obscured by dominant tissue-of-origin signals in pan-cancer RNA-seq data. The authors combine a compact autoencoder-based latent representation with stability-aware clustering to identify rare, reproducible subtypes, validating the approach with a pan-cancer negative control that shows tissue-driven clustering (Cramér’s ) and then focusing on KIRC to detect a rare C0 subtype at with prevalence and Jaccard . A differential-expression analysis yields a coherent marker signature for C0, underscoring the biological relevance of the subtype. This workflow provides a practical framework for unsupervised discovery of rare genomic subtypes in RNA-seq data and can be extended to richer datasets like TCGA to link transcriptional states with clinical outcomes and therapies.

Abstract

Unsupervised learning on high-dimensional RNA-seq data can reveal molecular subtypes beyond standard labels. We combine an autoencoder-based representation with clustering and stability analysis to search for rare but reproducible genomic subtypes. On the UCI "Gene Expression Cancer RNA-Seq" dataset (801 samples, 20,531 genes; BRCA, COAD, KIRC, LUAD, PRAD), a pan-cancer analysis shows clusters aligning almost perfectly with tissue of origin (Cramer's V = 0.887), serving as a negative control. We therefore reframe the problem within KIRC (n = 146): we select the top 2,000 highly variable genes, standardize them, train a feed-forward autoencoder (128-dimensional latent space), and run k-means for k = 2-10. While global indices favor small k, scanning k with a pre-specified discovery rule (rare < 10 percent and stable with Jaccard >= 0.60 across 20 seeds after Hungarian alignment) yields a simple solution at k = 5 (silhouette = 0.129, DBI = 2.045) with a rare cluster C0 (6.85 percent of patients) that is highly stable (Jaccard = 0.787). Cluster-vs-rest differential expression (Welch's t-test, Benjamini-Hochberg FDR) identifies coherent markers. Overall, pan-cancer clustering is dominated by tissue of origin, whereas a stability-aware within-cancer approach reveals a rare, reproducible KIRC subtype.

Paper Structure

This paper contains 22 sections, 7 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Dataset
  4. Pan-cancer negative control
  5. Preprocessing.
  6. Pan-cancer autoencoder and clustering.
  7. Evaluation.
  8. Within-cancer reframing: focus on KIRC
  9. Highly variable genes and scaling
  10. Autoencoder architecture and training
  11. Clustering, model selection and stability
  12. Within-KIRC clustering
  13. Cluster stability via Jaccard index
  14. Final clustering solution at $k=5$
  15. Cluster interpretability via differential expression
  16. ...and 7 more sections

Figures (9)

  • Figure 1: Pan-cancer k-means silhouette score as a function of $k$, computed on the autoencoder latent space ($Z$) for all 801 samples. The maximum occurs at $k=6$.
  • Figure 2: Example autoencoder training curve (KIRC): reconstruction MSE vs epoch for training and validation splits.
  • Figure 3: Within-KIRC k-means silhouette score as a function of $k$. The best score is at $k=2$, but rare subtypes emerge only at larger $k$.
  • Figure 4: Within-KIRC Davies--Bouldin index (DBI) as a function of $k$. Lower values indicate more compact and separated clusters.
  • Figure 5: Cluster sizes for KIRC at $k=5$. Cluster C0 (size 10, prevalence 6.8%) is highlighted in red as the rare subtype.
  • ...and 4 more figures