Histo-Genomic Knowledge Distillation For Cancer Prognosis From Histopathology Whole Slide Images

TL;DR

A novel Genome-informed Hyper-Attention Network, termed G-HANet, which is capable of effectively distilling the histo-genomic knowledge during training to elevate uni-modal whole slide image (WSI)-based inference for the first time and achieves competitive performance with genome-based and multi-modal methods.

Abstract

Histo-genomic multi-modal methods have recently emerged as a powerful paradigm, demonstrating significant potential for improving cancer prognosis. However, genome sequencing, unlike histopathology imaging, is still not widely accessible in underdeveloped regions, limiting the application of these multi-modal approaches in clinical settings. To address this, we propose a novel Genome-informed Hyper-Attention Network, termed G-HANet, which is capable of effectively distilling the histo-genomic knowledge during training to elevate uni-modal whole slide image (WSI)-based inference for the first time. Compared with traditional knowledge distillation methods (i.e., teacher-student architecture) in other tasks, our end-to-end model is superior in terms of training efficiency and learning cross-modal interactions. Specifically, the network comprises the cross-modal associating branch (CAB) and hyper-attention survival branch (HSB). Through the genomic data reconstruction from WSIs, CAB effectively distills the associations between functional genotypes and morphological phenotypes and offers insights into the gene expression profiles in the feature space. Subsequently, HSB leverages the distilled histo-genomic associations as well as the generated morphology-based weights to achieve the hyper-attention modeling of the patients from both histopathology and genomic perspectives to improve cancer prognosis. Extensive experiments are conducted on five TCGA benchmarking datasets and the results demonstrate that G-HANet significantly outperforms the state-of-the-art WSI-based methods and achieves competitive performance with genome-based and multi-modal methods. G-HANet is expected to be explored as a useful tool by the research community to address the current bottleneck of insufficient histo-genomic data pairing in the context of cancer prognosis and precision oncology.

Figures (5)

• Figure 1: Overview of the proposed genome-informed hyper-attention network (G-HANet). The WSIs and genomic data are formulated as bags of features $F^p$ and categorized according to the functions as $X^g$. In G-HANet, the Cross-modal Associating Branch distills the histo-genomic knowledge through genome reconstruction from WSIs. Afterward, with the assistance of the distilled knowledge, HSB achieves the hyper-attention modeling of the WSIs from both histopathology and genomic perspectives and conducts survival prediction. The genome data processing and reconstruction (framed by the red boxes) are only required in the training phase, thereby enabling the WSI-based inference.
• Figure 2: Overview of the hyper-attention module. It takes the transformed bag of features $F^p$ and association matrix $m$ as input and outputs the aggregated genome-informed features $f_{GI}$.
• Figure 3: Kaplan-Meier survival curves of TransMIL and our proposed G-HANet on five cancer datasets. High-risk and low-risk patients are indicated with red and blue, respectively. The shaded areas indicate the confidence intervals of each cohort. P-value $<$ 0.05 represents a significant statistical difference between the two groups.
• Figure 4: Violin plots of the Spearman Correlation coefficients of six genomic functions on five TCGA datasets. The mean values and corresponding standard variations are placed next to each plot.
• Figure 5: Histo-genomic association visualization from the G-HANet. Here, we particularly visualized the histo-genomic associations of the six different functional genes and presented the top four highlighted patches. Two WSIs from the GBMLGG and UCEC datasets were taken as examples.