Lifelong Whole Slide Image Analysis: Online Vision-Language Adaptation and Past-to-Present Gradient Distillation

TL;DR

ADaFGrad tackles lifelong learning for whole-slide image cancer subtyping by unifying online vision–language adaptation with past–to–present gradient distillation. It leverages a pathology vision–language foundation to align region-level features with text-based prototypes and preserves past knowledge via gradient-based replay, all within a rehearsal framework. Across six TCGA datasets, ADaFGrad achieves superior ACC, Masked ACC, and AUC with minimal training epochs and effective stability, outperforming state-of-the-art WSI continual-learning methods. The approach offers practical potential for distributed clinical deployment by enabling a single model to continually learn from new slides while retaining previous knowledge.

Abstract

Whole Slide Images (WSIs) play a crucial role in accurate cancer diagnosis and prognosis, as they provide tissue details at the cellular level. However, the rapid growth of computational tasks involving WSIs poses significant challenges. Given that WSIs are gigapixels in size, they present difficulties in terms of storage, processing, and model training. Therefore, it is essential to develop lifelong learning approaches for WSI analysis. In scenarios where slides are distributed across multiple institutes, we aim to leverage them to develop a unified online model as a computational tool for cancer diagnosis in clinical and hospital settings. In this study, we introduce ADaFGrad, a method designed to enhance lifelong learning for whole-slide image (WSI) analysis. First, we leverage pathology vision-language foundation models to develop a framework that enables interaction between a slide's regional tissue features and a predefined text-based prototype buffer. Additionally, we propose a gradient-distillation mechanism that mimics the gradient of a logit with respect to the classification-head parameters across past and current iterations in a continual-learning setting. We construct a sequence of six TCGA datasets for training and evaluation. Experimental results show that ADaFGrad outperforms both state-of-the-art WSI-specific and conventional continual-learning methods after only a few training epochs, exceeding them by up to +5.068% in the class-incremental learning scenario while exhibiting the least forgetting (i.e., retaining the most knowledge from previous tasks). Moreover, ADaFGrad surpasses its baseline by as much as +40.084% in accuracy, further demonstrating the effectiveness of the proposed modules.

Figures (7)

• Figure 1: Lifelong learning for WSI analysis.
• Figure 2: Overview of ADaFGrad: For a new task, class templates and irrelevant sentences serve as prototypes. These are encoded by a text encoder to extract prototype embeddings, which are stored in a prototype buffer. When a slide is processed, it is supervised by a cross-entropy loss $\mathcal{L}_{CE}$ and aligned with prototypes using the Online Vision-Language Adaptation loss $\mathcal{L}_{OVLA}$. Meanwhile, the gradient of the logit for the target label with respect to the classification head's parameters is stored and replayed for knowledge distillation through $\mathcal{L}_{PPGD}$.
• Figure 3: Illustration of OVLA.
• Figure 4: Distribution of the sequence of six TCGA datasets.
• Figure 5: Accuracy–Forgetting trade-off. The first row shows the ACC–FGT trade-off, and the second row shows the mACC–FGT trade-off.