HMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classification

TL;DR

The paper tackles fine-grained WSI classification by introducing HMIL, a hierarchical multi-instance learning framework that leverages coarse-to-fine label structure through class-wise attention and hierarchical alignment at both instance and bag levels. It combines offline and online feature extraction, supervised contrastive learning, and a curriculum-based dynamic weighting strategy to learn discriminative embeddings without heavy instance-level annotations. HMIL achieves state-of-the-art results on BRACS, PANDA, and CCC datasets, demonstrating improved accuracy, specificity, and AUC for both fine- and coarse-grained tasks, and shows robustness across histology and cytology modalities. The work advances WSI analysis by enforcing semantic consistency across label hierarchies and reducing annotation burdens, with potential for integration with pathology foundation models and domain adaptation strategies.

Abstract

Fine-grained classification of whole slide images (WSIs) is essential in precision oncology, enabling precise cancer diagnosis and personalized treatment strategies. The core of this task involves distinguishing subtle morphological variations within the same broad category of gigapixel-resolution images, which presents a significant challenge. While the multi-instance learning (MIL) paradigm alleviates the computational burden of WSIs, existing MIL methods often overlook hierarchical label correlations, treating fine-grained classification as a flat multi-class classification task. To overcome these limitations, we introduce a novel hierarchical multi-instance learning (HMIL) framework. By facilitating on the hierarchical alignment of inherent relationships between different hierarchy of labels at instance and bag level, our approach provides a more structured and informative learning process. Specifically, HMIL incorporates a class-wise attention mechanism that aligns hierarchical information at both the instance and bag levels. Furthermore, we introduce supervised contrastive learning to enhance the discriminative capability for fine-grained classification and a curriculum-based dynamic weighting module to adaptively balance the hierarchical feature during training. Extensive experiments on our large-scale cytology cervical cancer (CCC) dataset and two public histology datasets, BRACS and PANDA, demonstrate the state-of-the-art class-wise and overall performance of our HMIL framework. Our source code is available at https://github.com/ChengJin-git/HMIL.

Figures (6)

• Figure 1: Comparison among prior works and our proposed HMIL framework in fine-grained WSI analysis. Left: Conventional flat classification methods, which form fine-grained classification as a multi-class classification task. Middle: Prior hierarchical classification methods, which typically leverage detector-enriched instance feature for hierarchical classification. Right: Our HMIL framework relaxed the need for detectors, introducing hierarchical alignment at both instance and bag level to improve fine-grained classification.
• Figure 2: Overview of the proposed HMIL. We use fine-grained cervical cancer classification as an example. Patched WSI is fed into an offline feature extractor for the coarse features of the WSI, followed by an online feature re-embedding module that produces fine-grained feature. Subsequently, a dual-branch MIL architecture performs attention extraction and classification tasks at different hierarchical levels, with hierarchical alignment applied to instance and bag levels. Fully connected layers are then employed on top of the aggregated features in each branch to predict classification logits. Specifically, in the fine-grained branch, we incorporate supervised contrastive learning to further refine the feature representation. Finally, a dynamic weighting training strategy is incorporated to regulate the weights of these two branches throughout network training.
• Figure 3: Hierarchical mappings and sub-class distributions in BRACS brancati2022bracs, PANDA bulten2022artificial and our collected CCC datasets. The mappings are from the original datasets designed by pathologists.
• Figure 4: The class-wise AUC distribution of top-performing methods on BRACS (Top), PANDA (Middle), and CCC (Bottom) datasets.
• Figure 5: The t-SNE visualization on PANDA (top) and CCC (bottom) datasets. The upper section of each dataset displays coarse-grained classes, while the lower section showcases fine-grained classes.