Slide-based Graph Collaborative Training for Histopathology Whole Slide Image Analysis

TL;DR

A generic WSI analysis pipeline SlideGCD is proposed that can be adapted to any existing Multiple Instance Learning (MIL) frameworks and improve their performance, and concurrently initializes and refines the slide representation, as a guide for message passing in the slide-based graph.

Abstract

The development of computational pathology lies in the consensus that pathological characteristics of tumors are significant guidance for cancer diagnostics. Most existing research focuses on the inner-contextual information within each WSI yet ignores the possible inter-correlations between slides. As the development of tumors is a continuous process involving a series of histological, morphological, and genetic changes that accumulate over time, the similarities and differences between WSIs across various stages, grades, locations and patients should potentially contribute to the representation of WSIs and deserve to be taken into account in WSI modeling. To verify the advancement of introducing the slide inter-correlations into the representation learning of WSIs, we proposed a generic WSI analysis pipeline SlideGCD that can be adapted to any existing Multiple Instance Learning (MIL) frameworks and improve their performance. With the new paradigm, the prior knowledge of cancer development can participate in the end-to-end workflow, which concurrently initializes and refines the slide representation, as a guide for message passing in the slide-based graph. Extensive comparisons and experiments are conducted to validate the effectiveness and robustness of the proposed pipeline across 4 different tasks, including cancer subtyping, cancer staging, survival prediction, and gene mutation prediction, with 7 representative SOTA WSI analysis frameworks as backbones.

Figures (5)

• Figure 1: Motivation of our methods. (a) Multi-Instance Learning Frameworks. The main difference between MIL methods lies in the implementations of pooling operations. (b) Patch-based Graph Methods. The mainstream graph-based methods represent WSIs to graphs and transfer the WSI classification problem as graph classification. (c) Slide-based Graph Methods. SlideGCD conceptualizes the WSI classification problem as node classification to explore the inter-correlations between slides explicitly via GNNs.
• Figure 2: Illustration of the proposed SlideGCD framework. The framework consists of three phases, (a) Sample Pre-Processing. Each WSI is transformed into a sequence of patch embeddings following the universal settings of the MIL paradigm. (b) Patch Interaction. Slide embeddings are generated by the backbone MIL method for each sample. (c) Slide-based Graph Interaction. A slide-based graph is maintained and updated during each mini-batch training (colored indicators denote samples from the current mini-batch), then graph learning and knowledge distillation are conducted to explore slide-level correlations and align both branches.
• Figure 3: Illustration of the update of the Node Buffer in embedded space. Left: the initial state of the Node Buffer after the warmup, where the centers of each category are close. Right: the updated state after a mini-batch with 2 samples. The appended positive sample is too far from the center even compared with the farthest stored node thus it won't be updated into the buffer since we consider it as an amplified disturbance from the former network update. The appended negative sample is close enough to the center to replace the farthest negative node marked by stripes. At last, the outcome of the update is that the negative center shifts away from the positive center as we expected.
• Figure 4: Performance curves on the validation dataset in the five-fold cross-validation, where the error bar indicates the standard deviation of the metrics and the dashed line parallel to the horizontal axis represents the metrics that the baseline (DTFDMIL) can achieve.
• Figure 5: T-SNE visualizations of the overall distribution of the nodes from the node buffer and 20 highlighted hyperedges for TCGA-BLCA. Each point corresponds to a patient and its color goes deeper as the survival time increases. The stars in each sub-figure make up the hyperedge.