Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Skip and Skip: Segmenting Medical Images with Prompts

Jiawei Chen, Dingkang Yang, Yuxuan Lei, Lihua Zhang

TL;DR

This paper addresses medical image lesion segmentation under limited pixel-level annotations by leveraging image-level labels as coarse-grained prompts. It introduces SKS, a dual U-shaped two-stage architecture that uses a coarse-grained feature guidance branch to provide pyramid features to a fine-grained segmentation branch through three skips, enabling effective prompting. A three-layer Swin-T v2 pyramid extracts both coarse and fine features; a fusion rule $F_{fuse} = W \times (f^{i}_{\lambda} \oplus f^{k}_{\lambda}) + b$ enables the Connection's Artful Leap to couple information across levels. On the LITS dataset, SKS achieves a Dice score of $0.549$ with only $35$ finely annotated scans, outperforming pixel-annotation-only baselines like U-Net ($0.489$) and U-Net++ ($0.509$). These results demonstrate that exploiting existing image-level diagnoses can significantly reduce labeling burden while maintaining segmentation accuracy, with potential applicability to other medical imaging tasks.

Abstract

Most medical image lesion segmentation methods rely on hand-crafted accurate annotations of the original image for supervised learning. Recently, a series of weakly supervised or unsupervised methods have been proposed to reduce the dependence on pixel-level annotations. However, these methods are essentially based on pixel-level annotation, ignoring the image-level diagnostic results of the current massive medical images. In this paper, we propose a dual U-shaped two-stage framework that utilizes image-level labels to prompt the segmentation. In the first stage, we pre-train a classification network with image-level labels, which is used to obtain the hierarchical pyramid features and guide the learning of downstream branches. In the second stage, we feed the hierarchical features obtained from the classification branch into the downstream branch through short-skip and long-skip and get the lesion masks under the supervised learning of pixel-level labels. Experiments show that our framework achieves better results than networks simply using pixel-level annotations.

Skip and Skip: Segmenting Medical Images with Prompts

TL;DR

This paper addresses medical image lesion segmentation under limited pixel-level annotations by leveraging image-level labels as coarse-grained prompts. It introduces SKS, a dual U-shaped two-stage architecture that uses a coarse-grained feature guidance branch to provide pyramid features to a fine-grained segmentation branch through three skips, enabling effective prompting. A three-layer Swin-T v2 pyramid extracts both coarse and fine features; a fusion rule enables the Connection's Artful Leap to couple information across levels. On the LITS dataset, SKS achieves a Dice score of with only finely annotated scans, outperforming pixel-annotation-only baselines like U-Net () and U-Net++ (). These results demonstrate that exploiting existing image-level diagnoses can significantly reduce labeling burden while maintaining segmentation accuracy, with potential applicability to other medical imaging tasks.

Abstract

Most medical image lesion segmentation methods rely on hand-crafted accurate annotations of the original image for supervised learning. Recently, a series of weakly supervised or unsupervised methods have been proposed to reduce the dependence on pixel-level annotations. However, these methods are essentially based on pixel-level annotation, ignoring the image-level diagnostic results of the current massive medical images. In this paper, we propose a dual U-shaped two-stage framework that utilizes image-level labels to prompt the segmentation. In the first stage, we pre-train a classification network with image-level labels, which is used to obtain the hierarchical pyramid features and guide the learning of downstream branches. In the second stage, we feed the hierarchical features obtained from the classification branch into the downstream branch through short-skip and long-skip and get the lesion masks under the supervised learning of pixel-level labels. Experiments show that our framework achieves better results than networks simply using pixel-level annotations.
Paper Structure (11 sections, 3 figures, 1 table)

This paper contains 11 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. SKS Overview
  4. Feature Extraction
  5. Connection's Artful Leap
  6. Prompt and Segmentation
  7. EXPERIMENTS
  8. Experiment Setup
  9. Experiment Evaluation
  10. Ablation Study
  11. CONCLUSION

Figures (3)

  • Figure 1: Overview of SKS. This dual U-shaped two-stage framework is a multi-level pyramid structure containing a feature extraction module and a lesion segmentation module, in the former represented by the yellow part is the coarse-grained feature guidance branch, and the green part is the fine-grained feature student branch.
  • Figure 2: The structure of (a) the feature extraction module and (b) the lesion segmentation module.
  • Figure 3: Visualization of liver segmentation results. The images above are the 2D slices of the liver region with the lesions outlined in green boxes, and the images below show the segmentation results of our framework.