IRS: Incremental Relationship-guided Segmentation for Digital Pathology

TL;DR

The paper addresses the challenge of class-incremental, panoramic segmentation in digital pathology where temporally acquired data is partially labeled and models must generalize to OOD lesions. It introduces Incremental Relationship-guided Segmentation (IRS), combining an Incremental Universal Proposition Matrix to encode spatial-temporal class relations, a prompt-driven dynamic mixture-of-experts backbone with class- and scale-aware tokens, and relationship-guided distillation via an anatomy loss and cross-step consistency losses. The approach is demonstrated on a 24-class renal pathology dataset across multiple steps of data introduction, showing superior performance over baselines in 2-, 3-, and 4-step continual segmentation and providing insights from ablations on the anatomy-guided losses and backbone design. The proposed framework enhances domain generalization and offers a scalable, anatomy-informed solution for real-world digital pathology workflows where new phenotypes and lesions continuously emerge.

Abstract

Continual learning is rapidly emerging as a key focus in computer vision, aiming to develop AI systems capable of continuous improvement, thereby enhancing their value and practicality in diverse real-world applications. In healthcare, continual learning holds great promise for continuously acquired digital pathology data, which is collected in hospitals on a daily basis. However, panoramic segmentation on digital whole slide images (WSIs) presents significant challenges, as it is often infeasible to obtain comprehensive annotations for all potential objects, spanning from coarse structures (e.g., regions and unit objects) to fine structures (e.g., cells). This results in temporally and partially annotated data, posing a major challenge in developing a holistic segmentation framework. Moreover, an ideal segmentation model should incorporate new phenotypes, unseen diseases, and diverse populations, making this task even more complex. In this paper, we introduce a novel and unified Incremental Relationship-guided Segmentation (IRS) learning scheme to address temporally acquired, partially annotated data while maintaining out-of-distribution (OOD) continual learning capacity in digital pathology. The key innovation of IRS lies in its ability to realize a new spatial-temporal OOD continual learning paradigm by mathematically modeling anatomical relationships between existing and newly introduced classes through a simple incremental universal proposition matrix. Experimental results demonstrate that the IRS method effectively handles the multi-scale nature of pathological segmentation, enabling precise kidney segmentation across various structures (regions, units, and cells) as well as OOD disease lesions at multiple magnifications. This capability significantly enhances domain generalization, making IRS a robust approach for real-world digital pathology applications.

Figures (8)

• Figure 1: Illustration of continual learning challenges in renal pathology segmentation, depicting: (1) temporal data collection; (2) multi-scale comprehensive objects ranging from regions to cells; (3) partial annotations from different annotators; and (4) progression from normal anatomical structures to out-of-distribution (OOD) pathological lesions.
• Figure 2: This figure illustrates the transformation of complex clinical anatomical relationships within the kidney into a continual learning paradigm. (a) The kidney's anatomy showcases the spatial relationships among large-scale objects, including regions, functional units, cells, and lesions. (b) Previous continual learning approaches merely add new classes without establishing connections between new and old classes. (c) The proposed IRS method integrates anatomical knowledge into continual learning to preserve knowledge from old classes, even when only new classes are used during model training.
• Figure 3: This figure illustrates the design of the incremental universal proposition matrix as new classes are added in different settings. The proposed matrix is easily adaptable to large-scale objects within the continual learning paradigm.
• Figure 4: This figure illustrates the architecture of our proposed prompt-driven dynamic MoE network. The model integrates three different architectures using a self-attention mechanism and a dynamic MoE head to enhance segmentation capabilities for large-scale pathology segmentation in continual learning. Additionally, the model maintains a consistent architecture while dynamically accommodating an increasing number of segmentation classes.
• Figure 5: This figure showcases the key innovation of knowledge distillation in our proposed method. In subsequent steps, the model learns from new classes supervised by new labels, while old class tokens are utilized on new images to predict old classes. The new model distills knowledge from the previous model by maintaining similarity in old class tokens, latent features, decoder features, and prediction logits.