ScribFormer: Transformer Makes CNN Work Better for Scribble-based Medical Image Segmentation

TL;DR

ScribFormer tackles the challenge of learning global shape information from sparse scribble annotations in medical image segmentation by fusing local CNN features with global Transformer representations in a triple-branch architecture enhanced by an attention-guided CAM (ACAM) branch. It introduces feature coupling units to align CNN and Transformer features, and proposes ACAM-consistency losses to refine shallow activations under deep guidance. The method combines scribble supervision, dynamically mixed pseudo-labels, and ACAM-consistency to outperform state-of-the-art scribble-supervised methods and even rival some fully supervised approaches on three datasets (ACDC, MSCMRseg, HeartUII). Results show improved segmentation quality, robust ablations, and favorable inference efficiency compared with competing methods, illustrating the practical potential of Transformer-CNN hybrids in reducing annotation costs for medical image analysis.

Abstract

Most recent scribble-supervised segmentation methods commonly adopt a CNN framework with an encoder-decoder architecture. Despite its multiple benefits, this framework generally can only capture small-range feature dependency for the convolutional layer with the local receptive field, which makes it difficult to learn global shape information from the limited information provided by scribble annotations. To address this issue, this paper proposes a new CNN-Transformer hybrid solution for scribble-supervised medical image segmentation called ScribFormer. The proposed ScribFormer model has a triple-branch structure, i.e., the hybrid of a CNN branch, a Transformer branch, and an attention-guided class activation map (ACAM) branch. Specifically, the CNN branch collaborates with the Transformer branch to fuse the local features learned from CNN with the global representations obtained from Transformer, which can effectively overcome limitations of existing scribble-supervised segmentation methods. Furthermore, the ACAM branch assists in unifying the shallow convolution features and the deep convolution features to improve model's performance further. Extensive experiments on two public datasets and one private dataset show that our ScribFormer has superior performance over the state-of-the-art scribble-supervised segmentation methods, and achieves even better results than the fully-supervised segmentation methods. The code is released at https://github.com/HUANGLIZI/ScribFormer.

Figures (7)

• Figure 1: Performance comparison of segmentation results across various methods: (a) Input images, masks, and scribble annotations, (b) CNN-based fully-supervised method (UNet++ unet++), (c) CNN-based scribble-supervised method (UNet++), and (d) Our proposed ScribFormer.
• Figure 2: Overview of our proposed ScribFormer. The framework consists of the hybrid CNN-Transformer encoders, the CNN decoder, the Transformer decoder, and the attention-guided class activation map (ACAM) branch. Both the CNN prediction $y_{CNN}$ and the Transformer prediction $y_{Trans}$ are compared separately with the scribble annotations and the dynamically mixed pseudo label. Furthermore, the ACAM branch compares multi-scale ACAMs with ACAM-consistency.
• Figure 3: Schematic illustration of FCU (Feature Coupling Units). Due to inconsistency of feature dimensions of CNN and Transformers, feature maps from convolution blocks and patch embeddings from Transformer blocks are fused in the down-sample and up-sample blocks after the channel and spatial alignments.
• Figure 4: Schematic illustration of attention-guided class activation maps (ACAM) branch.
• Figure 5: Qualitative comparison between ScribFormer and other state-of-the-art methods on ACDC and MSCMRseg datasets. Subscripts F and S denote the segmentation models trained with fully-annotated masks or scribble annotations, respectively.