Multi-Granularity Vision Fastformer with Fusion Mechanism for Skin Lesion Segmentation

TL;DR

This work introduces VFFM-UNet, a lightweight U-shaped network that integrates Vision Fastformer with a Fusion Mechanism to tackle skin lesion segmentation. It employs Multi-Granularity Vision Fastformer to capture context at pixel, patch, and window scales and fuses them via Multi-Granularity and Channel Fusion to adapt to lesions with varying severity. Ablation studies show that both the VF and FM components contribute to improvements in accuracy with substantial reductions in parameters and GFLOPs, achieving state-of-the-art efficiency on ISIC2017, ISIC2018, and PH$^{2}$ datasets. The approach offers a practical balance between computational cost and segmentation performance, with strong generalization and potential applicability to other medical image segmentation tasks, albeit with some limitations on extremely irregular lesion boundaries.

Abstract

Background:Convolutional Neural Networks(CNN) and Vision Transformers(ViT) are the main techniques used in Medical image segmentation. However, CNN is limited to local contextual information, and ViT's quadratic complexity results in significant computational costs. At the same time, equipping the model to distinguish lesion boundaries with varying degrees of severity is also a challenge encountered in skin lesion segmentation. Purpose:This research aims to optimize the balance between computational costs and long-range dependency modelling and achieve excellent generalization across lesions with different degrees of severity. Methods:we propose a lightweight U-shape network that utilizes Vision Fastformer with Fusion Mechanism (VFFM-UNet). We inherit the advantages of Fastformer's additive attention mechanism, combining element-wise product and matrix product for comprehensive feature extraction and channel reduction to save computational costs. In order to accurately identify the lesion boundaries with varying degrees of severity, we designed Fusion Mechanism including Multi-Granularity Fusion and Channel Fusion, which can process the feature maps in the granularity and channel levels to obtain different contextual information. Results:Comprehensive experiments on the ISIC2017, ISIC2018 and PH2 datasets demonstrate that VFFM-UNet outperforms existing state-of-the-art models regarding parameter numbers, computational complexity and segmentation performance. In short, compared to MISSFormer, our model achieves superior segmentation performance while reducing parameter and computation costs by 101x and 15x, respectively. Conclusions:Both quantitative and qualitative analyses show that VFFM-UNet sets a new benchmark by reaching an ideal balance between parameter numbers, computational complexity, and segmentation performance compared to existing state-of-the-art models.

Figures (12)

• Figure 1: From left to right, the visualizations show the comparative experimental results on the ISIC2018, ISIC2017, and PH2 datasets. The X-axis represents the number of parameters (lower is better), while Y-axis represents mIoU (higher is better). The color depth represents computational complexity (GFLOPs, lighter is better).
• Figure 2: Accurately identifying the unclear boundaries of lesions requires certain contextual information.
• Figure 3: Lesions with different degrees of severity require different contextual information for identification of their boundaries.
• Figure 4: An overview of our proposed VFFM-UNet
• Figure 5: An overview of Vision Fastformer