A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends

TL;DR

This survey addresses the shift from hand-crafted features to deep learning in colorectal polyp segmentation, detailing traditional and modern model families, datasets, and evaluation practices. It systematically benchmarks 24 representative deep models (including CNN, Transformer, and hybrid architectures) across multiple datasets, with emphasis on scale-aware performance and temporal video segmentation. Key findings show Transformer-based approaches generally outperform CNNs, with top methods like DuAT, FeDNet, ESFPNet, Polyp-PVT, HSNet, and BDG-Net achieving strong results, while scale and cross-domain generalization remain challenging. The work highlights clinical deployment requirements, data/privacy considerations, and ethical issues, offering guidance for future research and providing open-access resources for reproducible evaluation.

Abstract

Early detection and assessment of polyps play a crucial role in the prevention and treatment of colorectal cancer (CRC). Polyp segmentation provides an effective solution to assist clinicians in accurately locating and segmenting polyp regions. In the past, people often relied on manually extracted lower-level features such as color, texture, and shape, which often had issues capturing global context and lacked robustness to complex scenarios. With the advent of deep learning, more and more outstanding medical image segmentation algorithms based on deep learning networks have emerged, making significant progress in this field. This paper provides a comprehensive review of polyp segmentation algorithms. We first review some traditional algorithms based on manually extracted features and deep segmentation algorithms, then detail benchmark datasets related to the topic. Specifically, we carry out a comprehensive evaluation of recent deep learning models and results based on polyp sizes, considering the pain points of research topics and differences in network structures. Finally, we discuss the challenges of polyp segmentation and future trends in this field. The models, benchmark datasets, and source code links we collected are all published at https://github.com/taozh2017/Awesome-Polyp-Segmentation.

Figures (6)

• Figure 1: A brief chronology of polyp segmentation. Before 2015, methods relied on hand-crafted features combined with machine learning algorithms. Since 2015, U-Net ronneberger2015u and FCN long2015fully have significantly advanced the development of deep learning techniques in polyp segmentation. More details can be found in Sec. \ref{['sec:models']}.
• Figure 2: Examples of images, ground truth maps, and edges in five polyp segmentation datasets, including (a) ETIS-LaribPolypDB silva2014toward, (b) CVC-ColonDB tajbakhsh2015automated, (c) CVC-ClinicDB bernal2015wm, (d) CVC-300 vazquez2017benchmark, and (e) Kvasir-SEG jha2020kvasir. In each dataset, the image, ground truth maps, and edges are shown from top to bottom.
• Figure 3: A comprehensive evaluation is conducted on 23 representative deep-learning models, including UNet ronneberger2015u, UNet++ zhou2018unet++, SFA fang2019selective, PraNet fan2020pranet, ACSNet zhang2020adaptive, MSEG huang2021hardnet, EU-Net patel2021enhanced, SANet wei2021shallow, MSNet zhao2021automatic, UACANet-S kim2021uacanet, UACANet-L kim2021uacanet, C2FNet sun2021context, DCRNet yin2022duplex, BDG-Net qiu2022bdg, CaraNet lou2022caranet, EFA-Net zhou2023edge, CFA-Net zhou2023cross, M2SNet zhao2023m, Polyp-PVT dong2021polyp, HSNet zhang2022hsnet, DuAT tang2023duat, ESFPNet chang2023esfpnet, and FeDNet su2023fednet, with SAM zhou2023can excluded. We report the average Dice and MAE values for each model across five datasets (i.e., ETIS-LaribPolypDB silva2014toward, CVC-ColonDB tajbakhsh2015automated, CVC-ClinicDB bernal2015wm, CVC-300 vazquez2017benchmark, and Kvasir-SEG jha2020kvasir). Please note that the models represented in the top left corner are better,i.e., they have larger Dice scores and smaller MAE values. In this context, the green triangles represent Transformer-based models, while the red diamonds signify CNN-based models.
• Figure 4: The PR curves and F-measures at different thresholds for 23 deep polyp segmentation models on five datasets from ETIS-LaribPolypDB silva2014toward, CVC-ColonDB tajbakhsh2015automated, CVC-ClinicDB bernal2015wm, CVC-300 vazquez2017benchmark, and Kvasir-SEG jha2020kvasir.
• Figure 5: Comparison results for 26 representative polyp segmentation models are shown in terms of $F_{\beta}$ (top) and $E_{\phi}$ (bottom).