A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models

TL;DR

This work tackles the challenge of subtyping Diffuse Large B-Cell Lymphoma (ABC vs GCB) from high-resolution whole-slide images by bypassing expensive molecular tests. It introduces a vision-transformer–based multi-modal teacher that learns from four WSIs modalities (HES and three IHC stains) and distills its knowledge into a mono-modal HES student, enabling accurate subtyping from a single modality. On 157 Lille patient WSIs, the mono-modal student achieves competitive accuracy and often surpasses several state-of-the-art WSI methods, with power-law scaling indicating further gains as more data are collected. External validation on a breast cancer dataset corroborates the approach's transferability, highlighting a practical path toward cheaper, faster, and robust lymphoma subtyping that could complement or reduce reliance on traditional IHC and RT-MLPA workflows.

Abstract

Determining lymphoma subtypes is a crucial step for better patient treatment targeting to potentially increase their survival chances. In this context, the existing gold standard diagnosis method, which relies on gene expression technology, is highly expensive and time-consuming, making it less accessibility. Although alternative diagnosis methods based on IHC (immunohistochemistry) technologies exist (recommended by the WHO), they still suffer from similar limitations and are less accurate. Whole Slide Image (WSI) analysis using deep learning models has shown promising potential for cancer diagnosis, that could offer cost-effective and faster alternatives to existing methods. In this work, we propose a vision transformer-based framework for distinguishing DLBCL (Diffuse Large B-Cell Lymphoma) cancer subtypes from high-resolution WSIs. To this end, we introduce a multi-modal architecture to train a classifier model from various WSI modalities. We then leverage this model through a knowledge distillation process to efficiently guide the learning of a mono-modal classifier. Our experimental study conducted on a lymphoma dataset of 157 patients shows the promising performance of our mono-modal classification model, outperforming six recent state-of-the-art methods. In addition, the power-law curve, estimated on our experimental data, suggests that with more training data from a reasonable number of additional patients, our model could achieve competitive diagnosis accuracy with IHC technologies. Furthermore, the efficiency of our framework is confirmed through an additional experimental study on an external breast cancer dataset (BCI dataset).

Figures (6)

• Figure 1: Knowledge Distillation for multi-to-mono modal WSI subtyping model.
• Figure 2: Our multi-modal architecture (teacher) takes as input a bag of sequences uniformly sampled from the same region across all WSI modalities (yellow squares). Its multi-modal features fusion mechanism produces a bag representation for multi-modal WSIs classification. Our mono-modal architecture is presented within the dotted bounding-box on the right.
• Figure 3: Our multi-modal features fusion mechanism.
• Figure 4: Performance comparisons of state-of-the-art methods obtained on the test set.
• Figure 5: Attention maps generated by our student model on partially tumoral regions.