Self-Contrastive Weakly Supervised Learning Framework for Prognostic Prediction Using Whole Slide Images

TL;DR

A novel three-part framework comprising of a convolutional network based tissue segmentation algorithm for region of interest delineation, a contrastive learning module for feature extraction, and a nested multiple instance learning classification module is proposed.

Abstract

We present a pioneering investigation into the application of deep learning techniques to analyze histopathological images for addressing the substantial challenge of automated prognostic prediction. Prognostic prediction poses a unique challenge as the ground truth labels are inherently weak, and the model must anticipate future events that are not directly observable in the image. To address this challenge, we propose a novel three-part framework comprising of a convolutional network based tissue segmentation algorithm for region of interest delineation, a contrastive learning module for feature extraction, and a nested multiple instance learning classification module. Our study explores the significance of various regions of interest within the histopathological slides and exploits diverse learning scenarios. The pipeline is initially validated on artificially generated data and a simpler diagnostic task. Transitioning to prognostic prediction, tasks become more challenging. Employing bladder cancer as use case, our best models yield an AUC of 0.721 and 0.678 for recurrence and treatment outcome prediction respectively.

Figures (5)

• Figure 1: Overview of ROIs utilized. The process of extracting ROIs from raw WSI involved either a tissue segmentation algorithm and/or pathologist's annotations. The annotations highlighted areas that were deemed prognostically significant for predicting outcome, while the algorithm provided masks that highlighted different tissue types. Subsets of tissue were extracted using urothelium and lamina propria. For magnification levels, the study explored two mono-scale approaches using 10x and 20x, as well as a multi-scale method using three magnifications (2.5x, 10x, 40x).
• Figure 2: Deep learning pipeline for prognostic outcome prediction. 1) A tissue segmentation is employed for delineating a ROI of choice $D$. Then,tiles are extracted from WSI regions for training an algorithm. 2) Contrastive learning is employed to learn representations of the tiles. 3) The representations are then used to train an AbMIL model that predicts the prognostic outcome. This approach compresses an end-to-end pipeline where the raw input image data is broken down and processed for predicting clinical outcome.
• Figure 3: Schematic representation of self-defined ROI generation. Guided by the segmentation mask of various tissue types, we apply diverse image processing morphological operations to define ROIs $D_{y}$ based on domain knowledge from expert pathologists. In the example displayed, we enlarge the urothelium and lamina propria masks applying dilation, limited by a distance parameter determined on clinical expertise. The resulting overlapping areas represent $D_{\text{BORDER}}$. We further delineate a subset of $D_{\text{BORDER}}$ by extracting only those areas where muscle is present within the same tissue section, aiming to represent the potential invasive front in $D_{\text{FRONT}}$.
• Figure 4: Box plot illustrating AUC performance variation across validation sets with different ROIs for 20x magnification. Notably, $D^{20x}_{\text{UROLP}}$ emerges as the top performer amongst the ROIs. White dots represent the average value $\mu$, and black diamonds represent outliers. The results show the mean $\mu$ and standard deviation $\sigma$ over 5 runs.
• Figure 5: Heatmap illustrating attention scores over a BCG-NR WSI from $S_{\text{EMC}}$. The heatmap provides insights into the ROIs where the attention is concentrated within the WSI, facilitating a better understanding of prediction dynamics and highlighting areas of significance for clinical interpretation.