Single color digital H&E staining with In-and-Out Net

TL;DR

A novel network, In-and-Out Net, designed explicitly for digital staining tasks is introduced, which efficiently transforms Reflectance Confocal Microscopy images into Hematoxylin and Eosin (H&E) stained images and enhances nuclei contrast in RCM images.

Abstract

Virtual staining streamlines traditional staining procedures by digitally generating stained images from unstained or differently stained images. While conventional staining methods involve time-consuming chemical processes, virtual staining offers an efficient and low infrastructure alternative. Leveraging microscopy-based techniques, such as confocal microscopy, researchers can expedite tissue analysis without the need for physical sectioning. However, interpreting grayscale or pseudo-color microscopic images remains a challenge for pathologists and surgeons accustomed to traditional histologically stained images. To fill this gap, various studies explore digitally simulating staining to mimic targeted histological stains. This paper introduces a novel network, In-and-Out Net, specifically designed for virtual staining tasks. Based on Generative Adversarial Networks (GAN), our model efficiently transforms Reflectance Confocal Microscopy (RCM) images into Hematoxylin and Eosin (H&E) stained images. We enhance nuclei contrast in RCM images using aluminum chloride preprocessing for skin tissues. Training the model with virtual H\&E labels featuring two fluorescence channels eliminates the need for image registration and provides pixel-level ground truth. Our contributions include proposing an optimal training strategy, conducting a comparative analysis demonstrating state-of-the-art performance, validating the model through an ablation study, and collecting perfectly matched input and ground truth images without registration. In-and-Out Net showcases promising results, offering a valuable tool for virtual staining tasks and advancing the field of histological image analysis.

Figures (6)

• Figure 1: Data collection and preprocessing. The images were collected with three channels simultaneously: reflectance channel (<495nm, grey box), SR101 channel (605±70nm, green box) and AO channel (525±50nm, red box). The stack images were conducted with surface extraction to form a single layer image. Reflectance channel images went through a inpainting step to remove the optical interference to generate the input images. Ground truth images were generated by AO channel and SR101 channel images.
• Figure 2: Model Architecture. a) Overall architecture. $G_H$, $G_E$ are generators, $D_H$, $D_E$ and $D_{out}$ are discriminators. $G_H$, $G_E$ and RGB concatenate layer forms $D_{out}$. b) Inner loop and outer loop. The red lines/arrows indicate active for training, while grey lines/arrows indicate inactive for training. c) Architecture for generators and discriminators.
• Figure 3: Examples of output images from different models. The generated images in columns from left to right are: Input RCM images, H&E ground truth, In-and-Out Net, pix2pixGAN, pix2pixHD, VSGD-Net and RestainNet loss. The scale bar is 50 µm.
• Figure 4: Violin plots comparing model metrics. We illustrate the differences between metrics obtained from the In-and-Out model and those from other models. The violin plots depict these differences.
• Figure 5: Examples of the ablation study. Ablation 1 removes In-and-Out training, resulting in slightly faded colors and absurd structure boundaries. Ablation 2 removes $D_{out}$, leading to faded colors. Ablation 3 removes $D_H \& D_E$, causing the image to appear blurry. The scale bar is 50 µm.