Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeMM-GAN: A Multimodal Generative Model Conditioned on Histopathology Images and Clinical Descriptions for Gene Expression Profile Generation

Francesca Pia Panaccione, Carlo Sgaravatti, Pietro Pinoli

TL;DR

GeMM-GAN introduces a multimodal generative framework that conditions gene expression synthesis on histopathology image patches and clinical descriptions, addressing data scarcity and privacy in transcriptomics. By fusing image and text via FiLM, a Transformer-based patch encoder, and bidirectional cross-attention to produce conditioning signals for a WGAN-GP, the method generates biologically coherent gene expression profiles. Across TCGA data, GeMM-GAN achieves superior distributional fidelity and downstream predictive utility, with notable gains in disease-type and primary-site classification and substantial preservation of gene-gene co-expression. The work demonstrates the value of integrating morphology and clinical context for realistic synthetic omics data, with potential to enable privacy-preserving research and cross-modal analyses in precision medicine.

Abstract

Biomedical research increasingly relies on integrating diverse data modalities, including gene expression profiles, medical images, and clinical metadata. While medical images and clinical metadata are routinely collected in clinical practice, gene expression data presents unique challenges for widespread research use, mainly due to stringent privacy regulations and costly laboratory experiments. To address these limitations, we present GeMM-GAN, a novel Generative Adversarial Network conditioned on histopathology tissue slides and clinical metadata, designed to synthesize realistic gene expression profiles. GeMM-GAN combines a Transformer Encoder for image patches with a final Cross Attention mechanism between patches and text tokens, producing a conditioning vector to guide a generative model in generating biologically coherent gene expression profiles. We evaluate our approach on the TCGA dataset and demonstrate that our framework outperforms standard generative models and generates more realistic and functionally meaningful gene expression profiles, improving by more than 11\% the accuracy on downstream disease type prediction compared to current state-of-the-art generative models. Code will be available at: https://github.com/francescapia/GeMM-GAN

GeMM-GAN: A Multimodal Generative Model Conditioned on Histopathology Images and Clinical Descriptions for Gene Expression Profile Generation

TL;DR

GeMM-GAN introduces a multimodal generative framework that conditions gene expression synthesis on histopathology image patches and clinical descriptions, addressing data scarcity and privacy in transcriptomics. By fusing image and text via FiLM, a Transformer-based patch encoder, and bidirectional cross-attention to produce conditioning signals for a WGAN-GP, the method generates biologically coherent gene expression profiles. Across TCGA data, GeMM-GAN achieves superior distributional fidelity and downstream predictive utility, with notable gains in disease-type and primary-site classification and substantial preservation of gene-gene co-expression. The work demonstrates the value of integrating morphology and clinical context for realistic synthetic omics data, with potential to enable privacy-preserving research and cross-modal analyses in precision medicine.

Abstract

Biomedical research increasingly relies on integrating diverse data modalities, including gene expression profiles, medical images, and clinical metadata. While medical images and clinical metadata are routinely collected in clinical practice, gene expression data presents unique challenges for widespread research use, mainly due to stringent privacy regulations and costly laboratory experiments. To address these limitations, we present GeMM-GAN, a novel Generative Adversarial Network conditioned on histopathology tissue slides and clinical metadata, designed to synthesize realistic gene expression profiles. GeMM-GAN combines a Transformer Encoder for image patches with a final Cross Attention mechanism between patches and text tokens, producing a conditioning vector to guide a generative model in generating biologically coherent gene expression profiles. We evaluate our approach on the TCGA dataset and demonstrate that our framework outperforms standard generative models and generates more realistic and functionally meaningful gene expression profiles, improving by more than 11\% the accuracy on downstream disease type prediction compared to current state-of-the-art generative models. Code will be available at: https://github.com/francescapia/GeMM-GAN
Paper Structure (24 sections, 8 equations, 2 figures, 2 tables)

This paper contains 24 sections, 8 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Method
  5. Single-Modal Networks
  6. Multimodal Fusion
  7. Generative Model
  8. Implementation Details
  9. Data
  10. Preprocessing
  11. Embedding models and Hyperparameters
  12. Experiments
  13. Evaluation metrics
  14. Unsupervised Metrics
  15. Precision and Recall
  16. ...and 9 more sections

Figures (2)

  • Figure 1: The architecture of our generative model. The input patches and textual descriptions are embedded with the Image and Text Encoder, respectively. These embeddings are combined in the Multimodal Fusion network to condition a WGAN-GP.
  • Figure 2: Preprocessing of the WSIs with Otsu Thresholding to extract valid image patches that contain at least 20% of tissue content