A Value Mapping Virtual Staining Framework for Large-scale Histological Imaging

TL;DR

A loss function based on the value mapping constraint is proposed to ensure the accuracy of virtual coloring between different pathological modalities, termed the Value Mapping Generative Adversarial Network (VM-GAN), and a confidence-based tiling method is presented to address the challenge of boundary inconsistency arising from patch-wise processing.

Abstract

The emergence of virtual staining technology provides a rapid and efficient alternative for researchers in tissue pathology. It enables the utilization of unlabeled microscopic samples to generate virtual replicas of chemically stained histological slices, or facilitate the transformation of one staining type into another. The remarkable performance of generative networks, such as CycleGAN, offers an unsupervised learning approach for virtual coloring, overcoming the limitations of high-quality paired data required in supervised learning. Nevertheless, large-scale color transformation necessitates processing large field-of-view images in patches, often resulting in significant boundary inconsistency and artifacts. Additionally, the transformation between different colorized modalities typically needs further efforts to modify loss functions and tune hyperparameters for independent training of networks. In this study, we introduce a general virtual staining framework that is adaptable to various conditions. We propose a loss function based on the value mapping constraint to ensure the accuracy of virtual coloring between different pathological modalities, termed the Value Mapping Generative Adversarial Network (VM-GAN). Meanwhile, we present a confidence-based tiling method to address the challenge of boundary inconsistency arising from patch-wise processing. Experimental results on diverse data with varying staining protocols demonstrate that our method achieves superior quantitative indicators and improved visual perception.

Figures (5)

• Figure 1: Architecture of the proposed VM-GAN model. (a) The main structure of the model, where the total loss function includes the adversarial loss, the cycle consistency loss, and the value loss. $\mathit{v}$(·) represents the value channel of the image. G and D correspond to the generator and discriminator respectively. (b) The process of converting from RGB color space to HSV space and extracting the value channel.
• Figure 2: Confidence tiling scheme for large-scale samples. During the zero-padding process in the bottom right corner, the filling interval for each small patches is $m$ pixel. $n$ represents the size of small patches and $l$ represents the size of the $WSI_{in}$. After processing with the weight matrix, the small patches are summed up to obtain the $WSI_{out}$. The example in the lower half of the figure demonstrates in detail the process of how the weight matrix functions.
• Figure 3: H$\&$E virtual staining results on autofluorescence images. (a) and (b) display the autofluorescence slides and their consecutive registered H$\&$E stained slices. (c), (d), and (e) present the H$\&$E virtual staining results obtained by the original CycleGAN Zhu2017, the method proposed by Wang et al. Wang2020Virtual, and the UTOM method Li2021. (f) exhibits the H$\&$E virtual staining results obtained using our method. Enlarged images are provided in the first two rows with a scale bar of 60 $\mu$m, and whole slice images (WSIs) of different samples are shown in the last four rows with a scale bar of 500 $\mu$m. Our method effectively avoids the color reversal phenomenon in virtual staining, achieving performance comparable to that of UTOM. AF: autofluorescence.
• Figure 4: Virtual re-staining from H$\&$E to IHC(Ki67). (a) shows the input real H$\&$E-stained slice, with the real Ki67 stained slice in the top left corner. (b), (c), (d), and (e) display the virtual IHC(Ki67) stained results obtained using the original CycleGAN, the method by Wang et al., the original UTOM, and UTOM with corrected mask, respectively. (f) represents the virtual IHC(Ki67) stained results obtained using our proposed VM-GAN method. Red arrows indicate coloring errors and inconsistent artifacts between patches. Our method effectively addresses the problem of inconsistent staining of similar spatial structures.
• Figure 5: Virtual re-staining from IHC(Cc10) to IHC(CD31) on the ANHIR lung lesion dataset. (a) is the initial Cc10 stained slice. (b) to (f) correspond to the virtual re-staining results obtained using different methods, including the original CycleGAN, the method by Wang et al., the original UTOM, UTOM with corrected mask, and our method. The scale bar represents 800 $\mu$m for the WSIs and 150 $\mu$m for the enlarged images. (g) is the real CD31 stained slice used as the reference. (h) and (i) represent the line profiles of the black and white dashed lines in the corresponding WSIs, respectively. This experiment proves that our method effectively addresses the problem of inconsistent staining contrast between patches and their surrounding. UTOMcor: UTOM with corrected mask.