HATs: Hierarchical Adaptive Taxonomy Segmentation for Panoramic Pathology Image Analysis

TL;DR

The paper tackles panoramic segmentation of kidney pathology by introducing Hierarchical Adaptive Taxonomy Segmentation (HATs), which encodes anatomical relationships across regions, functional units, and cells using a hierarchical taxonomy matrix $M_t \in \mathbb{R}^{n\times n}$ and a hierarchical scale matrix informed by area-rate knowledge. A token-based EfficientSAM with a dynamic token bank stores class-aware tokens $T_c \in \mathbb{R}^{n\times d}$ and scale tokens $T_s \in \mathbb{R}^{4\times d}$ to enable weak-prompt semantic segmentation and cross-scale awareness, with a taxonomy loss $L_{hats}$ guiding relationships and a scale-weighted loss $S$. The method is validated on a 15-class kidney dataset compiled from Regions, Functional Units, and Cells across NEPTUNE, HuBMAP, and nephrectomy sources, using a two-phase training regime and 512×512 patches; results show superior Dice scores compared to multiple baselines and demonstrate the benefits of the hierarchical matrices and token-based dynamics. The work provides an open-source implementation and points to future enhancements by hybridizing CNN and transformer backbones to further improve segmentation across scales.

Abstract

Panoramic image segmentation in computational pathology presents a remarkable challenge due to the morphologically complex and variably scaled anatomy. For instance, the intricate organization in kidney pathology spans multiple layers, from regions like the cortex and medulla to functional units such as glomeruli, tubules, and vessels, down to various cell types. In this paper, we propose a novel Hierarchical Adaptive Taxonomy Segmentation (HATs) method, which is designed to thoroughly segment panoramic views of kidney structures by leveraging detailed anatomical insights. Our approach entails (1) the innovative HATs technique which translates spatial relationships among 15 distinct object classes into a versatile "plug-and-play" loss function that spans across regions, functional units, and cells, (2) the incorporation of anatomical hierarchies and scale considerations into a unified simple matrix representation for all panoramic entities, (3) the adoption of the latest AI foundation model (EfficientSAM) as a feature extraction tool to boost the model's adaptability, yet eliminating the need for manual prompt generation in conventional segment anything model (SAM). Experimental findings demonstrate that the HATs method offers an efficient and effective strategy for integrating clinical insights and imaging precedents into a unified segmentation model across more than 15 categories. The official implementation is publicly available at https://github.com/hrlblab/HATs.

Figures (4)

• Figure 1: Knowledge transformation from kidney anatomy to a hierarchical taxonomy tree. This figure demonstrates the transformation of intricate clinical anatomical relationships within the kidney into a hierarchical taxonomy tree. (a) Pathologists examine histopathology in accordance with kidney anatomy. (b) This study revisits kidney anatomy using a hierarchical semantic taxonomy for panoramic segmentation , covering 15 classes across regions, units, and cells. The tree incorporates spatial relationships into a semi-supervised learning paradigm and uses hierarchical scale information as prior knowledge to weigh the relationship between classes.
• Figure 2: hierarchical taxonomy learning -- This figure highlights the key innovation of the proposed taxonomy learning strategy. (a) A hierarchical taxonomy matrix is modeled from anatomical relationships to Aristotle's logic theory in pathological image segmentation. (b) A novel taxonomy loss function is designed to operationalize the affirmative and negatory relationships from hierarchical taxonomy matrix during the training process. (c) We further encode a hierarchical scale matrix to illustrate the strength of the relationship between different objects in kidney anatomy.
• Figure 3: Dynamic EfficientSAM with token bank -- This figure visualizes the architecture of our proposed token-based dynamic EfficientSAM. Key components include a dynamic token bank with class-aware and scale-aware tokens, a token-guided imageViT encoder, a mask decoder, and a dynamic head network. This architecture leverages AI fundation model by fine-tuning with weak tokens, liberating the model from the need for pixel-level image prompts.
• Figure 4: Validation qualitative results -- This figure shows the qualitative results of different approaches. The proposed method achieved superior panoramic kidney pathology segmentation on 15 classes range regions to cells with fewer false positives, false negatives, and morphological errors.