Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PARF-Net: integrating pixel-wise adaptive receptive fields into hybrid Transformer-CNN network for medical image segmentation

Xu Ma, Mengsheng Chen, Junhui Zhang, Lijuan Song, Fang Du, Zhenhua Yu

TL;DR

A new method PARF-Net is developed to integrate convolutions of Pixel-wise Adaptive Receptive Fields (Conv-PARF) into hybrid Network for medical image segmentation, surpassing existing methods by a large margin.

Abstract

Convolutional neural networks (CNNs) excel in local feature extraction while Transformers are superior in processing global semantic information. By leveraging the strengths of both, hybrid Transformer-CNN networks have become the major architectures in medical image segmentation tasks. However, existing hybrid methods still suffer deficient learning of local semantic features due to the fixed receptive fields of convolutions, and also fall short in effectively integrating local and long-range dependencies. To address these issues, we develop a new method PARF-Net to integrate convolutions of Pixel-wise Adaptive Receptive Fields (Conv-PARF) into hybrid Network for medical image segmentation. The Conv-PARF is introduced to cope with inter-pixel semantic differences and dynamically adjust convolutional receptive fields for each pixel, thus providing distinguishable features to disentangle the lesions with varying shapes and scales from the background. The features derived from the Conv-PARF layers are further processed using hybrid Transformer-CNN blocks under a lightweight manner, to effectively capture local and long-range dependencies, thus boosting the segmentation performance. By assessing PARF-Net on four widely used medical image datasets including MoNuSeg, GlaS, DSB2018 and multi-organ Synapse, we showcase the advantages of our method over the state-of-the-arts. For instance, PARF-Net achieves 84.27% mean Dice on the Synapse dataset, surpassing existing methods by a large margin.

PARF-Net: integrating pixel-wise adaptive receptive fields into hybrid Transformer-CNN network for medical image segmentation

TL;DR

A new method PARF-Net is developed to integrate convolutions of Pixel-wise Adaptive Receptive Fields (Conv-PARF) into hybrid Network for medical image segmentation, surpassing existing methods by a large margin.

Abstract

Convolutional neural networks (CNNs) excel in local feature extraction while Transformers are superior in processing global semantic information. By leveraging the strengths of both, hybrid Transformer-CNN networks have become the major architectures in medical image segmentation tasks. However, existing hybrid methods still suffer deficient learning of local semantic features due to the fixed receptive fields of convolutions, and also fall short in effectively integrating local and long-range dependencies. To address these issues, we develop a new method PARF-Net to integrate convolutions of Pixel-wise Adaptive Receptive Fields (Conv-PARF) into hybrid Network for medical image segmentation. The Conv-PARF is introduced to cope with inter-pixel semantic differences and dynamically adjust convolutional receptive fields for each pixel, thus providing distinguishable features to disentangle the lesions with varying shapes and scales from the background. The features derived from the Conv-PARF layers are further processed using hybrid Transformer-CNN blocks under a lightweight manner, to effectively capture local and long-range dependencies, thus boosting the segmentation performance. By assessing PARF-Net on four widely used medical image datasets including MoNuSeg, GlaS, DSB2018 and multi-organ Synapse, we showcase the advantages of our method over the state-of-the-arts. For instance, PARF-Net achieves 84.27% mean Dice on the Synapse dataset, surpassing existing methods by a large margin.
Paper Structure (29 sections, 8 equations, 4 figures, 6 tables)

This paper contains 29 sections, 8 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CNN-based Models
  4. Transformer-based Models
  5. Hybrid Transformer-CNN based Models
  6. Methods
  7. Convolutions of pixel-wise adaptive receptive fields
  8. Hybrid Transformer-CNN modules
  9. Overall architecture of PARF-Net
  10. Model optimization
  11. Datasets
  12. MoNuSeg dataset kumar2017datasetkumar2019multi
  13. GlaS dataset sirinukunwattana2017gland
  14. DSB2018 dataset
  15. Synapse dataset landman2015miccai
  16. ...and 14 more sections

Figures (4)

  • Figure 1: The model architecture of proposed PARF-Net. A U-shaped network that integrates convolutions of pixel-wise adaptive receptive fields (Conv-PARF) and hybrid Transformer-CNN modules is employed in PARF-Net. The Conv-PARF layers utilize spatial attention mechanism to dynamically extract spatial local information with the pixel-wise adaptive receptive fields. The hybrid modules are responsible for learning and fusing local and long-range dependencies, providing high-quality low-resolution features.
  • Figure 2: Visualization of segmentation results on the MoNuSeg, GlaS and DSB2018 datasets. For each dataset, two medical images accompanied with the ground truth segmentation masks and predicted results are shown.
  • Figure 3: Visualization of segmentation results on the multi-organ Synapse dataset.
  • Figure 4: Visualization of the activation maps corresponding to different receptive fields.