Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Standardized Framework For Evaluating Gene Expression Generative Models

Andrea Rubbi, Andrea Giuseppe Di Francesco, Mohammad Lotfollahi, Pietro Liò

Abstract

The rapid development of generative models for single-cell gene expression data has created an urgent need for standardised evaluation frameworks. Current evaluation practices suffer from inconsistent metric implementations, incomparable hyperparameter choices, and a lack of biologically-grounded metrics. We present Generated Genetic Expression Evaluator (GGE), an open-source Python framework that addresses these challenges by providing a comprehensive suite of distributional metrics with explicit computation space options and biologically-motivated evaluation through differentially expressed gene (DEG)-focused analysis and perturbation-effect correlation, enabling standardized reporting and reproducible benchmarking. Through extensive analysis of the single-cell generative modeling literature, we identify that no standardized evaluation protocol exists. Methods report incomparable metrics computed in different spaces with different hyperparameters. We demonstrate that metric values vary substantially depending on implementation choices, highlighting the critical need for standardization. GGE enables fair comparison across generative approaches and accelerates progress in perturbation response prediction, cellular identity modeling, and counterfactual inference.

A Standardized Framework For Evaluating Gene Expression Generative Models

Abstract

The rapid development of generative models for single-cell gene expression data has created an urgent need for standardised evaluation frameworks. Current evaluation practices suffer from inconsistent metric implementations, incomparable hyperparameter choices, and a lack of biologically-grounded metrics. We present Generated Genetic Expression Evaluator (GGE), an open-source Python framework that addresses these challenges by providing a comprehensive suite of distributional metrics with explicit computation space options and biologically-motivated evaluation through differentially expressed gene (DEG)-focused analysis and perturbation-effect correlation, enabling standardized reporting and reproducible benchmarking. Through extensive analysis of the single-cell generative modeling literature, we identify that no standardized evaluation protocol exists. Methods report incomparable metrics computed in different spaces with different hyperparameters. We demonstrate that metric values vary substantially depending on implementation choices, highlighting the critical need for standardization. GGE enables fair comparison across generative approaches and accelerates progress in perturbation response prediction, cellular identity modeling, and counterfactual inference.
Paper Structure (42 sections, 5 equations, 1 figure, 4 tables)

This paper contains 42 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work: The Evaluation Landscape
  3. Generative models for single-cell perturbation data
  4. Survey of Evaluation Practices
  5. Theoretical Foundations
  6. Problem Formulation
  7. Distributional metrics: Optimal Transport, MMD, and Energy
  8. The Space Question: A Theoretical Analysis
  9. Data normalization and transformation.
  10. Biological interpretation of distributional metrics.
  11. An effective framework.
  12. The GGE Framework
  13. Architecture and Design Principles
  14. Explicit Configuration.
  15. Universal Space Support.
  16. ...and 27 more sections

Figures (1)

  • Figure 1: GGE is a unified framework for the evaluation of generated genetic expression profiles. GGE aims to standardise the otherwise heterogeneous set of evaluation protocols proposed by the multitude of generative models implemented for this task.