MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models

TL;DR

MIPHEI-ViT addresses the cost and accessibility barrier of multiplex immunofluorescence by predicting 16 mIF channels from standard H&E using a U-Net framework augmented with ViT foundation encoders. It achieves strong marker-level and cell-level performance across internal and external datasets, with ablations showing LoRA-fine-tuned ViTMatte encoders as the best configuration and no GAN required for optimal results. The study emphasizes cell-level validation over pixel metrics, demonstrating robust cross-domain generalization to datasets with different staining protocols. This approach enables large-scale, cell-type-aware analyses of historical H&E cohorts, enabling biomarker discovery and hypothesis generation in oncology while highlighting markers that remain challenging to predict.

Abstract

Histopathological analysis is a cornerstone of cancer diagnosis, with Hematoxylin and Eosin (H&E) staining routinely acquired for every patient to visualize cell morphology and tissue architecture. On the other hand, multiplex immunofluorescence (mIF) enables more precise cell type identification via proteomic markers, but has yet to achieve widespread clinical adoption due to cost and logistical constraints. To bridge this gap, we introduce MIPHEI (Multiplex Immunofluorescence Prediction from H&E), a U-Net-inspired architecture that integrates state-of-the-art ViT foundation models as encoders to predict mIF signals from H&E images. MIPHEI targets a comprehensive panel of markers spanning nuclear content, immune lineages (T cells, B cells, myeloid), epithelium, stroma, vasculature, and proliferation. We train our model using the publicly available ORION dataset of restained H&E and mIF images from colorectal cancer tissue, and validate it on two independent datasets. MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.88 for Pan-CK, 0.57 for CD3e, 0.56 for SMA, 0.36 for CD68, and 0.30 for CD20, substantially outperforming both a state-of-the-art baseline and a random classifier for most markers. Our results indicate that our model effectively captures the complex relationships between nuclear morphologies in their tissue context, as visible in H&E images and molecular markers defining specific cell types. MIPHEI offers a promising step toward enabling cell-type-aware analysis of large-scale H&E datasets, in view of uncovering relationships between spatial cellular organization and patient outcomes.

Figures (5)

• Figure 1: Preprocessing pipeline:(a) H&E and mIF images are aligned using Valis valis. Tissue regions are then selected via Otsu thresholding on H&E, and a trained CNN filters misaligned tiles caused by the restaining and acquisition process. (b) Autofluorescence is subtracted from mIF images, followed by DAPI-based nuclei segmentation and nuclei dilation to approximate cell boundaries. (c) Pseudo-labels are generated by computing per-cell mean marker expression and applying GMM clustering to define marker positivity, determining labels (e.g., CD3e+).
• Figure 2: MIPHEI architecture: A U-Net-inspired model, based on VitMatte, is trained to predict mIF images from H&E, using the H-optimus-0 ViT foundation model, Tanh activation, and a custom weighted MSE loss to correct for the unbalanced distribution across markers.
• Figure 3: Dataset Overview:(a) Distribution of tissue samples and tiles across datasets and splits. (b) 2D UMAP visualization of H-Optimized-0 embeddings, highlighting domain shifts across our datasets. (c) Normalized cell type distribution across datasets and splits.
• Figure 4: Prediction pipeline and prediction visualization:(a) Inference pipeline: mIF images are first generated using a trained U-Net model. Predicted single-cell data are then extracted by averaging predicted mIF signals within each nucleus, using nuclei masks from an external segmentation model. Finally a cell classifier, trained on validation set cells, predicts cell types. (b) Prediction examples from our best model: Predicted mIF images and cell types (shown as colored cell boundaries) are compared to target mIF images and annotated cell types from the same restained tissue section in the Orion (CD3e, CD8a) and HEMIT (Pan-CK, CD3) datasets. (c) IMMUcan large-area visualization: Nuclei predictions on H&E alongside clustered nuclei from the corresponding consecutive mIF sections.
• Figure 5: Evaluation analysis:(a) Orion test set: Performance comparison across all Orion markers between MIPHEI, HEMIT*, HEMIT (all trained on Orion train set) and a stratified random model predicting classes based on cell type proportions. Markers are grouped by general functions, with hierarchical relationships indicated by arrows. Cell classification model is a logistic regression model trained on Orion validation cells. (b) HEMIT dataset: Comparison between MIPHEI (trained on Orion), the HEMIT model (trained on HEMIT train set) and a stratified random model. Cell classification model is a logistic regression: trained on 5% of training cells for MIPHEI, and all available training cells for the HEMIT model. (c) IMMUcan (consecutive sections): Pearson correlation and regression plots (with linear fits) between predicted and pseudo-labeled cell type counts from 17k tiles using MIPHEI.