Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers

TL;DR

Polyp-PVT introduces a pyramid vision transformer-based encoder for robust polyp segmentation and couples it with three lightweight modules—CFM for high-level fusion, CIM for extracting camouflage cues in low-level features, and SAM for integrating multi-scale features via non-local and graph-based processing. The approach achieves state-of-the-art accuracy across five challenging datasets and demonstrates strong generalization to unseen centers, including ColonDB and ETIS, as well as competitive performance in video polyp segmentation. Key findings show that explicit cross-level feature fusion and noise suppression significantly improve boundary delineation and resilience to appearance changes, small polyps, and rotations. The work suggests that transformer-based backbones, when paired with targeted fusion and attention mechanisms, can substantially advance clinical endoscopy image analysis with improved reliability and generalization.

Abstract

Most polyp segmentation methods use CNNs as their backbone, leading to two key issues when exchanging information between the encoder and decoder: 1) taking into account the differences in contribution between different-level features and 2) designing an effective mechanism for fusing these features. Unlike existing CNN-based methods, we adopt a transformer encoder, which learns more powerful and robust representations. In addition, considering the image acquisition influence and elusive properties of polyps, we introduce three standard modules, including a cascaded fusion module (CFM), a camouflage identification module (CIM), and a similarity aggregation module (SAM). Among these, the CFM is used to collect the semantic and location information of polyps from high-level features; the CIM is applied to capture polyp information disguised in low-level features, and the SAM extends the pixel features of the polyp area with high-level semantic position information to the entire polyp area, thereby effectively fusing cross-level features. The proposed model, named Polyp-PVT, effectively suppresses noises in the features and significantly improves their expressive capabilities. Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations (e.g., appearance changes, small objects, rotation) than existing representative methods. The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.

Figures (12)

• Figure 1: The segmentation examples of our model and SANet wei2021shallow with different challenge cases, e.g., camouflage ($1^{st}$ and $2^{nd}$ rows) and image acquisition influence ($3^{rd}$ row). The images from top to bottom are from ClinicDB bernal2015wm, ETIS silva2014toward, and ColonDB tajbakhsh2015automated, which show that our model has better generalization ability.
• Figure 2: Framework of our Polyp-PVT, which consists of a pyramid vision transformer (PVT) (a) as the encoder network, (b) cascaded fusion module (CFM) for fusing the high-level feature, (c) camouflage identification module (CIM) to filter out the low-level information, and (d) similarity aggregation module (SAM) for integrating the high- and low-level features for the final output.
• Figure 3: Details of the introduced SAM. It is composed of GCN and non-local, which extend the pixel features of polyp regions with high-level semantic location cues to the entire region.
• Figure 4: Loss curves under different training parameter settings.
• Figure 5: Evaluation of model generalization ability. We provide the max Dice results on ColonDB and ETIS.