Discovery of a Hematopoietic Manifold in scGPT Yields a Method for Extracting Performant Algorithms from Biological Foundation Model Internals

TL;DR

It is reported that scGPT internally encodes a compact hematopoietic manifold with significant developmental branch structure, validated on a strict non-overlap Tabula Sapiens external panel and confirmed via frozen-head zero-shot transfer to an independent multi-donor immune panel.

Abstract

We report the discovery and extraction of a compact hematopoietic algorithm from the single-cell foundation model scGPT, to our knowledge the first biologically useful, competitive algorithm extracted from a foundation model via mechanistic interpretability. We show that scGPT internally encodes a compact hematopoietic manifold with significant developmental branch structure, validated on a strict non-overlap Tabula Sapiens external panel and confirmed via frozen-head zero-shot transfer to an independent multi-donor immune panel. To isolate this geometry, we introduce a general three-stage extraction method consisting of direct operator export from frozen attention weights, a lightweight learned adaptor, and a task-specific readout, producing a standalone algorithm without target-dataset retraining. In 88-split donor-holdout benchmarks against scVI, Palantir, DPT, CellTypist, PCA, and raw-expression baselines, the extracted algorithm achieves the strongest pseudotime-depth ordering and leads on key subtype endpoints (CD4/CD8 AUROC 0.867, mono/macro AUROC 0.951). Compared to standard probing of frozen scGPT embeddings with a 3-layer MLP, the extracted head is BH-significantly better on 6/8 classification endpoints while completing a full 12-split evaluation campaign 34.5x faster with approximately 1000x fewer trainable parameters. The exported operator compresses from three pooled attention heads to a single head without statistically significant loss, and further to a rank-64 surrogate. Mechanistic interpretability of the compact operator reveals a concentrated four-factor core explaining 66.2% of ablation impact, with factors resolving into explicit T/lymphoid, B/plasma, granulocytic, and monocyte/macrophage gene programs. A supplementary second-manifold validation (intercellular communication geometry) confirms that the extraction method generalizes beyond hematopoiesis.

Figures (17)

• Figure 1: Hematopoietic manifold discovered in scGPT attention geometry (internal panel). Distinct branch structure emerges around stem/progenitor, erythroid, granulocytic, monocyte/macrophage, and lymphoid/T-cell regions.
• Figure 2: Holdout transfer comparison. The strict non-overlap external panel passes all gates, while the lung panel fails robust transfer despite strong global association.
• Figure 3: Robust-V2 split-level pseudotime deltas versus H65 cell head (method minus cell head). All alternatives are below zero.
• Figure 4: Fixed-probe leave-one-factor-out ablation for the rank-64 compact operator. (a) Total pooled ablation impact for the 12 most impactful factors. (b) Cumulative concentration curve---the top four factors explain $66.2\%$ of total ablation impact. (c) Per-task drop magnitudes; asterisks denote BH-significant loss versus the intact model.
• Figure 5: Core-subset recovery as a fraction of intact rank-64 performance across the exhaustive 15-subset interaction sweep. Green borders mark the best subset for each endpoint. No subset closes the gap to the intact model (all remain BH-significant); task-specialized circuitry is visible---mono/macro is well-recovered by the pair {f01, f03} alone (0.95$\times$), while branch and stage require all four factors. Best subsets per endpoint are listed in Supplementary Table \ref{['tab:rank64_core_interaction_best_subsets']}.