Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EP-SAM: Weakly Supervised Histopathology Segmentation via Enhanced Prompt with Segment Anything

Joonhyeon Song, Seohwan Yun, Seongho Yoon, Joohyeok Kim, Sangmin Lee

TL;DR

This work proposes a novel approach beyond supervised learning for effective pathological image analysis, addressing the challenge of limited robust labeled data by combining class activation map and Segment Anything Model (SAM)-based pseudo-labeling in a weakly supervised semantic segmentation (WSSS) model.

Abstract

This work proposes a novel approach beyond supervised learning for effective pathological image analysis, addressing the challenge of limited robust labeled data. Pathological diagnosis of diseases like cancer has conventionally relied on the evaluation of morphological features by physicians and pathologists. However, recent advancements in compute-aided diagnosis (CAD) systems are gaining significant attention as diagnostic support tools. Although the advancement of deep learning has improved CAD significantly, segmentation models typically require large pixel-level annotated dataset, and such labeling is expensive. Existing studies not based on supervised approaches still struggle with limited generalization, and no practical approach has emerged yet. To address this issue, we present a weakly supervised semantic segmentation (WSSS) model by combining class activation map and Segment Anything Model (SAM)-based pseudo-labeling. For effective pretraining, we adopt the SAM-a foundation model that is pretrained on large datasets and operates in zero-shot configurations using only coarse prompts. The proposed approach transfer enhanced Attention Dropout Layer's knowledge to SAM, thereby generating pseudo-labels. To demonstrate the superiority of the proposed method, experimental studies are conducted on histopathological breast cancer datasets. The proposed method outperformed other WSSS methods across three datasets, demonstrating its efficiency by achieving this with only 12GB of GPU memory during training. Our code is available at : https://github.com/QI-NemoSong/EP-SAM

EP-SAM: Weakly Supervised Histopathology Segmentation via Enhanced Prompt with Segment Anything

TL;DR

This work proposes a novel approach beyond supervised learning for effective pathological image analysis, addressing the challenge of limited robust labeled data by combining class activation map and Segment Anything Model (SAM)-based pseudo-labeling in a weakly supervised semantic segmentation (WSSS) model.

Abstract

This work proposes a novel approach beyond supervised learning for effective pathological image analysis, addressing the challenge of limited robust labeled data. Pathological diagnosis of diseases like cancer has conventionally relied on the evaluation of morphological features by physicians and pathologists. However, recent advancements in compute-aided diagnosis (CAD) systems are gaining significant attention as diagnostic support tools. Although the advancement of deep learning has improved CAD significantly, segmentation models typically require large pixel-level annotated dataset, and such labeling is expensive. Existing studies not based on supervised approaches still struggle with limited generalization, and no practical approach has emerged yet. To address this issue, we present a weakly supervised semantic segmentation (WSSS) model by combining class activation map and Segment Anything Model (SAM)-based pseudo-labeling. For effective pretraining, we adopt the SAM-a foundation model that is pretrained on large datasets and operates in zero-shot configurations using only coarse prompts. The proposed approach transfer enhanced Attention Dropout Layer's knowledge to SAM, thereby generating pseudo-labels. To demonstrate the superiority of the proposed method, experimental studies are conducted on histopathological breast cancer datasets. The proposed method outperformed other WSSS methods across three datasets, demonstrating its efficiency by achieving this with only 12GB of GPU memory during training. Our code is available at : https://github.com/QI-NemoSong/EP-SAM

Paper Structure

This paper contains 27 sections, 4 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Explicit Visual Prompting in Computer Vision
  4. WSSS in Histopathology
  5. Effective Prompts for SAM
  6. Transferring Knowledge to SAM for WSSS
  7. Fine-tuning SAM for Downstream Task
  8. Method
  9. Overview
  10. Initial Mask Generation Phase
  11. Enhanced Attention Dropout Layer
  12. Post-processing
  13. SAM Mask Generation Phase
  14. Pixel-level Entropy based Prompting Module (PEPM)
  15. Preliminary SAM Mask Decoder Fine-tuning
  16. ...and 12 more sections

Figures (7)

  • Figure 1: Overview of our proposed method.
  • Figure 2: Various CAMs for Camelyon17.
  • Figure 3: Overall scenario in inference phase.
  • Figure 4: Zero-shot results of SAM with respect to the iterative changes in masks. Progressive refinement is observed as training progresses, with the white tumor region becoming closer to the ground truth as iterations increase.
  • Figure 5: Qualitative comparison between our proposed method and MIL-based methods across all datasets. Left: Camelyon16, Camelyon17 datasets. Right: internal dataset.
  • ...and 2 more figures