Transforming H&E images into IHC: A Variance-Penalized GAN for Precision Oncology

TL;DR

This study proposes an advanced deep learning-based image translation framework to generate high-fidelity IHC images from H&E-stained tissue samples, enabling cost-effective and scalable HER2 assessment and marks a significant step toward AI-driven precision oncology.

Abstract

The overexpression of the human epidermal growth factor receptor 2 (HER2) in breast cells is a key driver of HER2-positive breast cancer, a highly aggressive subtype requiring precise diagnosis and targeted therapy. Immunohistochemistry (IHC) is the standard technique for HER2 assessment but is costly, labor-intensive, and highly dependent on antibody selection. In contrast, hematoxylin and eosin (H&E) staining, a routine histopathological procedure, offers broader accessibility but lacks HER2 specificity. This study proposes an advanced deep learning-based image translation framework to generate high-fidelity IHC images from H&E-stained tissue samples, enabling cost-effective and scalable HER2 assessment. By modifying the loss function of pyramid pix2pix, we mitigate mode collapse, a fundamental limitation in generative adversarial networks (GANs), and introduce a novel variance-based penalty that enforces structural diversity in generated images. Our model particularly excels in translating HER2-positive (IHC 3+) images, which have remained challenging for existing methods. Quantitative evaluations on the overall BCI dataset reveal that our approach outperforms baseline models, achieving a peak signal-to-noise ratio (PSNR) of 22.16, a structural similarity index (SSIM) of 0.47, and a Fréchet Inception Distance (FID) of 346.37. In comparison, the pyramid pix2pix baseline attained PSNR 21.15, SSIM 0.43, and FID 516.75, while the standard pix2pix model yielded PSNR 20.74, SSIM 0.44, and FID 472.6. These results affirm the superior fidelity and realism of our generated IHC images. Beyond medical imaging, our model exhibits superior performance in general image-to-image translation tasks, showcasing its potential across multiple domains. This work marks a significant step toward AI-driven precision oncology, offering a reliable and efficient alternative to traditional HER2 diagnostics.

Figures (16)

• Figure 1: Different types of IHC images based on levels of HER2 expression in the BCI dataset liu2022bci. The higher the HER2 expression, the greater the corresponding label.
• Figure 2: Different techniques for HER2 detection.
• Figure 3: The framework of the proposed architecture for generating high-fidelty IHC images from H&E images. The relative variance between input images and generated images in a batch is incorporated into the overall loss of pyramid pix2pix liu2022bci.
• Figure 4: Computation of the proposed variance loss. After computing batch-wise pixel variance for real and generated tensors, the absolute difference is calculated at each pixel location, followed by averaging to obtain the final variance loss.
• Figure 5: Representative image patches from the BCI dataset liu2022bci, showcasing hematoxylin and eosin (H&E) stained images alongside their corresponding immunohistochemistry (IHC) images. The dataset provides paired samples for deep learning-based image translation and biomarker assessment.