Rethinking Boundary Detection in Deep Learning-Based Medical Image Segmentation

TL;DR

CTO addresses the boundary segmentation challenge in medical image analysis by fusing local CNN features, global StitchViT representations, and explicit Sobel-based boundary guidance within a boundary-guided decoder. The dual-stream encoder learns complementary local and global cues, while the boundary-extracted (BEM) and boundary-injected (BIM) modules provide deep supervision to sharpen boundaries without extra annotations. Across seven diverse MedISeg datasets, CTO achieves state-of-the-art accuracy with competitive model complexity, demonstrating the value of explicit boundary priors and hybrid architectures for precise, boundary-aware medical segmentation. The work also includes extensive ablations and comparisons with large vision models, and discusses limitations and avenues for future work, including 3D extension and domain-shift robustness.

Abstract

Medical image segmentation is a pivotal task within the realms of medical image analysis and computer vision. While current methods have shown promise in accurately segmenting major regions of interest, the precise segmentation of boundary areas remains challenging. In this study, we propose a novel network architecture named CTO, which combines Convolutional Neural Networks (CNNs), Vision Transformer (ViT) models, and explicit edge detection operators to tackle this challenge. CTO surpasses existing methods in terms of segmentation accuracy and strikes a better balance between accuracy and efficiency, without the need for additional data inputs or label injections. Specifically, CTO adheres to the canonical encoder-decoder network paradigm, with a dual-stream encoder network comprising a mainstream CNN stream for capturing local features and an auxiliary StitchViT stream for integrating long-range dependencies. Furthermore, to enhance the model's ability to learn boundary areas, we introduce a boundary-guided decoder network that employs binary boundary masks generated by dedicated edge detection operators to provide explicit guidance during the decoding process. We validate the performance of CTO through extensive experiments conducted on seven challenging medical image segmentation datasets, namely ISIC 2016, PH2, ISIC 2018, CoNIC, LiTS17, and BTCV. Our experimental results unequivocally demonstrate that CTO achieves state-of-the-art accuracy on these datasets while maintaining competitive model complexity. The codes have been released at: https://github.com/xiaofang007/CTO.

Figures (8)

• Figure 1: Visualization comparisons of medical image segmentation results obtained from ViT dosovitskiy2020image, CNNs-ViT hatamizadeh2022swin, boundary detection operatorr kanopoulos1988design, and our proposed CTO. The results demonstrate that our method, which incorporates the explicit boundary detection, can achieve a significant improvement, especially in boundary areas, indicating the importance and effectiveness of the boundary detection operator in MedISeg. The red dashed bounding box highlights the core idea of our method. Samples are from the ISIC dataset gutman2016skincodella2019skin.
• Figure 2: (a) Illustration of our proposed Convolution, Transformer, and Operator (CTO), which follows an encoder-decoder paradigm. The dual-stream encoder network consists of a mainstream CNNs stream in the left side of (b) and an auxiliary ViT stream in the right side of (b), and the outputs of these two streams are fused together and used as the input of the decoder network. The boundary-guided decoder network in (c) employs an explicit boundary-enhanced module to guide its learning process. Specifically, the boundary detection operator is utilized to generate a self-generated boundary mask via the boundary-enhanced module that is then incorporated into the decoder network to enhance the boundary learning capacity and improve the segmentation accuracy. CTO integrates CNNs, ViT, and boundary detection into a unified framework. The red circle denotes the effective receptive field of the CNNs model, while the colored boxes represent the sliding windows of the ViT model.
• Figure 3: Basic module of Res2Net gao2019res2net.
• Figure 4: Illustration of Stitch-Vit. First, the channels of the feature map are divided into different heads. Then, "stitch" operation is performed to sample the feature map using different stitch rates in different heads. Finally, the sampled feature maps are concatenated and transformed to the original feature map. The red box highlights the stitch rates of our method, and the blue points reveal the sampling positions.
• Figure 5: Visualized results of the ablation study on ISIC 2018 codella2019skin. "w/" denotes with the corresponding module implementation.