Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all

Ihab Bendidi, Shawn Whitfield, Kian Kenyon-Dean, Hanene Ben Yedder, Yassir El Mesbahi, Emmanuel Noutahi, Alisandra K. Denton

TL;DR

A novel biologically motivated evaluation framework and a hierarchy of perturbation analysis tasks for comparing the performance of pretrained foundation models to each other and to more classical techniques of learning from transcriptomics data are presented.

Abstract

Understanding the relationships among genes, compounds, and their interactions in living organisms remains limited due to technological constraints and the complexity of biological data. Deep learning has shown promise in exploring these relationships using various data types. However, transcriptomics, which provides detailed insights into cellular states, is still underused due to its high noise levels and limited data availability. Recent advancements in transcriptomics sequencing provide new opportunities to uncover valuable insights, especially with the rise of many new foundation models for transcriptomics, yet no benchmark has been made to robustly evaluate the effectiveness of these rising models for perturbation analysis. This article presents a novel biologically motivated evaluation framework and a hierarchy of perturbation analysis tasks for comparing the performance of pretrained foundation models to each other and to more classical techniques of learning from transcriptomics data. We compile diverse public datasets from different sequencing techniques and cell lines to assess models performance. Our approach identifies scVI and PCA to be far better suited models for understanding biological perturbations in comparison to existing foundation models, especially in their application in real-world scenarios.

Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all

TL;DR

A novel biologically motivated evaluation framework and a hierarchy of perturbation analysis tasks for comparing the performance of pretrained foundation models to each other and to more classical techniques of learning from transcriptomics data are presented.

Abstract

Understanding the relationships among genes, compounds, and their interactions in living organisms remains limited due to technological constraints and the complexity of biological data. Deep learning has shown promise in exploring these relationships using various data types. However, transcriptomics, which provides detailed insights into cellular states, is still underused due to its high noise levels and limited data availability. Recent advancements in transcriptomics sequencing provide new opportunities to uncover valuable insights, especially with the rise of many new foundation models for transcriptomics, yet no benchmark has been made to robustly evaluate the effectiveness of these rising models for perturbation analysis. This article presents a novel biologically motivated evaluation framework and a hierarchy of perturbation analysis tasks for comparing the performance of pretrained foundation models to each other and to more classical techniques of learning from transcriptomics data. We compile diverse public datasets from different sequencing techniques and cell lines to assess models performance. Our approach identifies scVI and PCA to be far better suited models for understanding biological perturbations in comparison to existing foundation models, especially in their application in real-world scenarios.

Paper Structure

This paper contains 37 sections, 39 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Transcriptomics Data.
  4. Transcriptomics Foundation Models.
  5. Perturbation Analysis and Benchmarks.
  6. Perturbation Hierarchy of Evaluation Metrics
  7. Data Integration and Batch Effect Reduction
  8. Latent Space Linear Separability of Known Perturbations
  9. Perturbation Consistency
  10. Latent Space Direct Organization
  11. Zero-Shot Retrieval of Known Biological Relationships
  12. Linear Interpretability of Latent Space
  13. Benchmark Experimental Setup
  14. Results
  15. Current Foundation models do not generalize to perturbation related tasks
  16. ...and 22 more sections

Figures (2)

  • Figure 1: Known biological relationship recall scores for Replogle2022-lx and L1000 Assay for scVI, trained using different gene distributions. Different datasets and sequencing approaches benefit from different gene distributions.
  • Figure 2: Training scVI on different amounts of samples, using distinct batches and perturbations, before evaluating known biological relationship retrieval. scVI is robust to very low data regime, and shows strong data scaling laws on single cell data.