Histo-Miner: Deep learning based tissue features extraction pipeline from H&E whole slide images of cutaneous squamous cell carcinoma

TL;DR

Histo-Miner, a deep learning-based pipeline designed for the analysis of skin WSIs, is deployed to predict cSCC patient response to immunotherapy based on pre-treatment WSIs from 45 patients, and is designed to allow for its use on other cancer types and on other training datasets.

Abstract

Recent advancements in digital pathology have enabled comprehensive analysis of Whole-Slide Images (WSI) from tissue samples, leveraging high-resolution microscopy and computational capabilities. Despite this progress, there is a lack of labeled datasets and open source pipelines specifically tailored for analysis of skin tissue. Here we propose Histo-Miner, a deep learning-based pipeline for analysis of skin WSIs and generate two datasets with labeled nuclei and tumor regions. We develop our pipeline for the analysis of patient samples of cutaneous squamous cell carcinoma (cSCC), a frequent non-melanoma skin cancer. Utilizing the two datasets, comprising 47,392 annotated cell nuclei and 144 tumor-segmented WSIs respectively, both from cSCC patients, Histo-Miner employs convolutional neural networks and vision transformers for nucleus segmentation and classification as well as tumor region segmentation. Performance of trained models positively compares to state of the art with multi-class Panoptic Quality (mPQ) of 0.569 for nucleus segmentation, macro-averaged F1 of 0.832 for nucleus classification and mean Intersection over Union (mIoU) of 0.907 for tumor region segmentation. From these predictions we generate a compact feature vector summarizing tissue morphology and cellular interactions, which can be used for various downstream tasks. Here, we use Histo-Miner to predict cSCC patient response to immunotherapy based on pre-treatment WSIs from 45 patients. Histo-Miner identifies percentages of lymphocytes, the granulocyte to lymphocyte ratio in tumor vicinity and the distances between granulocytes and plasma cells in tumors as predictive features for therapy response. This highlights the applicability of Histo-Miner to clinically relevant scenarios, providing direct interpretation of the classification and insights into the underlying biology.

Figures (5)

• Figure 1: Overview of Histo-Miner pipeline.The pipeline uses WSI from cSCC patient as input. (a) & (b)[id=lu]During inference, WSI images are tiled into patches and undergo pre-processing pipelines (see Methods). After pre-processing, SCC Segmenter performs tumor region binary segmentation on processed patches and SCC Hovernet segments and classifies cell nuclei. [id=lu]Images are processed via two different pre-processing pipelines that involve downsampling, tiling and color normalization. After the first pre-processing pipeline, resulting images are input to SCC Segmenter for tumor region binary segmentation. In parallel, images after the second pre-processing pipeline are input to SCC Hovernet to segment and classify cell nuclei. (c) Using the output of SCC Segmenter, cell classification is refined by adding a new cell class: non-neoplastic epithelial. This last result is saved in a json text file. A visualization of resulting annotations is provided in Fig. \ref{['fig2']}. (d) Using refined segmentation and classification of nuclei, together with segmentation of tumor regions, we calculate features that describe the tissue organization. Example features include e.g. percentage of lymphocytes in the vicinity of the tumor and average closest distance between tumor and lymphocyte cells. A list of all 317 calculated features is provided in Supplementary Data.
• Figure 2: Predictions of SCC Hovernet and SCC Segmenter models.The different images correspond to different steps of the Histo-Miner pipeline as depicted in Fig. \ref{['fig1']}. 2 WSIs of 2 different patients from 2 different cohorts are shown. The H&E staining differs between the slides showing varying hues of blue and pink. After predicting the tumor area with SCC Segmenter, Histo-Miner segments and then classifies cells into five different classes: granulocytes, lymphocytes, plasma cells, stromal cells, tumor cells. Using tumor segmentation, tumor cells detected outside tumor regions are re-classified as non-neoplastic. The cell nuclei segmentation and classification illustrate sample organization at tissue level (3x zoom), or at cell level (75x zoom). Based on segmentation results Histo-Miner calculates features describing cell-level and tissue-level tumor organization. Also in the case of damaged sample (one part of the tumor is missing in the WSI on the right), the model is not hallucinating segmentation of the remaining parts of the sample.
• Figure 3: Visualization of samples from NucSeg and TumSeg training datasets.(a) Overview of both datasets (b) Visualization of NucSeg training dataset. 47,392 cell nuclei from 1,707 H&E non-overlapping patches were segmented and classified. (c) Visualization of TumSeg training dataset. Tumor region are segmented by 2 experts on 144 WSIs.
• Figure 4: Validation of SCC Hovernet.(a) Panoptic Quality for each cell class of SCC Hovernet, light $\text{CellViT}_{256}$ and heavy CellViT-SAM-B, all trained on NucSeg. mPQ is the average of PQ for all classes. SCC Hovernet outperforms $\text{CellViT}_{256}$ for all classes and outperforms CellViT-SAM-B general mPQ performance. CellViT-SAM-B outperforms SCC Hovernet on general classification performance. Taking into account segmentation, detection, classification tasks and weight of the models, SCC Hovernet is the preferred option. (b) Confusion matrix from SCC Hovernet prediction on the validation set. The most representative class, tumor cells, is accurately classified 94% of the time. The worst classification accuracy is of plasma cells: 74%. (c) Examples of validation via immunohistochemistry showing H&E and cell-type predictions (left and middle column) and staining for cell type markers of the same are in a consecutive section (right column) Mypo=Myeloperoxidase. (d) Comparison of [id=lu]manual cell type [id=lu]quantification[id=lu]counts in immunohistochemistry slides (x-axis[id=lu]; Mypo = Myeloperoxidase) and computationally predicted cells (y-axis) in H&E slides.
• Figure 5: Histo-Miner tested in a clinical scenario: prediction of CPI treatment response.(a) To test the clinical utility of Histo-Miner we assembled and processed WSIs from in total n=45 cSCC patients before checkpoint inhibition (CPI) therapy. (b) Mean ROC curve for the classifier keeping 19 best features and its standard error. The classifier cross-validation folds ROC curves as well as the standard deviation of the mean ROC curve are shown in Supplementary Fig. 6. (c) On the left, box plots of the 4 best features, p-value was calculated using Mann-Whitney U test. On the right, visualization of representative cases for each of the best features. For distance visualization we hide cell classes other than plasma cells and granulocytes.