Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HEMIT: H&E to Multiplex-immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator

Chang Bian, Beth Philips, Tim Cootes, Martin Fergie

TL;DR

HEMIT tackles the challenge of translating traditional H&E histology to multiplex mIHC images by introducing a publicly available cellularly aligned dataset and a dual-branch Pix2Pix generator that fuses CNN-based spatial detail with Swin Transformer–driven multi-scale context. The dataset pairs H&E and multi-marker mIHC (DAPI, CD3, panCK) from the same tissue section, enabling precise cell-to-cell translation and robust supervision. The proposed architecture, including a Feature Map Fusion module and cross-attention between branches, achieves superior image translation performance on SSIM, Pearson correlation, and PSNR, and is validated through downstream cell-level analyses using QuPath and Stardist. This work provides a valuable resource and benchmark for stain translation research and paves the way for biomarker inference from conventional H&E slides in oncology research.

Abstract

Computational analysis of multiplexed immunofluorescence histology data is emerging as an important method for understanding the tumour micro-environment in cancer. This work presents HEMIT, a dataset designed for translating Hematoxylin and Eosin (H&E) sections to multiplex-immunohistochemistry (mIHC) images, featuring DAPI, CD3, and panCK markers. Distinctively, HEMIT's mIHC images are multi-component and cellular-level aligned with H&E, enriching supervised stain translation tasks. To our knowledge, HEMIT is the first publicly available cellular-level aligned dataset that enables H&E to multi-target mIHC image translation. This dataset provides the computer vision community with a valuable resource to develop novel computational methods which have the potential to gain new insights from H&E slide archives. We also propose a new dual-branch generator architecture, using residual Convolutional Neural Networks (CNNs) and Swin Transformers which achieves better translation outcomes than other popular algorithms. When evaluated on HEMIT, it outperforms pix2pixHD, pix2pix, U-Net, and ResNet, achieving the highest overall score on key metrics including the Structural Similarity Index Measure (SSIM), Pearson correlation score (R), and Peak signal-to-noise Ratio (PSNR). Additionally, downstream analysis has been used to further validate the quality of the generated mIHC images. These results set a new benchmark in the field of stain translation tasks.

HEMIT: H&E to Multiplex-immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator

TL;DR

HEMIT tackles the challenge of translating traditional H&E histology to multiplex mIHC images by introducing a publicly available cellularly aligned dataset and a dual-branch Pix2Pix generator that fuses CNN-based spatial detail with Swin Transformer–driven multi-scale context. The dataset pairs H&E and multi-marker mIHC (DAPI, CD3, panCK) from the same tissue section, enabling precise cell-to-cell translation and robust supervision. The proposed architecture, including a Feature Map Fusion module and cross-attention between branches, achieves superior image translation performance on SSIM, Pearson correlation, and PSNR, and is validated through downstream cell-level analyses using QuPath and Stardist. This work provides a valuable resource and benchmark for stain translation research and paves the way for biomarker inference from conventional H&E slides in oncology research.

Abstract

Computational analysis of multiplexed immunofluorescence histology data is emerging as an important method for understanding the tumour micro-environment in cancer. This work presents HEMIT, a dataset designed for translating Hematoxylin and Eosin (H&E) sections to multiplex-immunohistochemistry (mIHC) images, featuring DAPI, CD3, and panCK markers. Distinctively, HEMIT's mIHC images are multi-component and cellular-level aligned with H&E, enriching supervised stain translation tasks. To our knowledge, HEMIT is the first publicly available cellular-level aligned dataset that enables H&E to multi-target mIHC image translation. This dataset provides the computer vision community with a valuable resource to develop novel computational methods which have the potential to gain new insights from H&E slide archives. We also propose a new dual-branch generator architecture, using residual Convolutional Neural Networks (CNNs) and Swin Transformers which achieves better translation outcomes than other popular algorithms. When evaluated on HEMIT, it outperforms pix2pixHD, pix2pix, U-Net, and ResNet, achieving the highest overall score on key metrics including the Structural Similarity Index Measure (SSIM), Pearson correlation score (R), and Peak signal-to-noise Ratio (PSNR). Additionally, downstream analysis has been used to further validate the quality of the generated mIHC images. These results set a new benchmark in the field of stain translation tasks.
Paper Structure (13 sections, 7 equations, 6 figures, 2 tables)

This paper contains 13 sections, 7 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. HEMIT Dataset
  3. Data Collection
  4. Data Preprocessing
  5. Proposed Method
  6. Architecture
  7. Swin Transformer-based Auxiliary Branch
  8. Feature Map Fusion Module
  9. Loss Functions
  10. Experiments
  11. Benchmark Results
  12. Downstream Analysis
  13. Conclusion

Figures (6)

  • Figure 1: Overview of dataset processing pipeline of the HEMIT dataset.
  • Figure 2: Alignment comparison of HEMIT and other datasets: (a) Visualization of registered image pairs of HEMIT dataset. (b) Visualization of the misaligned image pairs in BCI dataset liu2022bci and MIST li2023adaptive dataset which used consecutive sectioning. Yellow arrows point to the edges of tissues in HER2 IHC images, and blue arrows point to the edges of tissues in H&E images to enhance visualization of the misalignment.
  • Figure 3: Overall structure of the proposed generator.
  • Figure 4: Visualization of different methods on HEMIT dataset. DAPI is shown in blue, panCK in red, and CD3 in green. Below each patch, four zoomed-in regions are displayed to provide a more detailed view.
  • Figure 5: Downstream analysis pipeline: Real and generated virtual mIHC images are analyzed using the Qupath StarDist model for cell detection. Following cell detection, the Otsu thresholding method is applied to identify positive cell groups. Finally, measurements are taken for evaluation purposes.
  • ...and 1 more figures