Colorectal Polyp Segmentation in the Deep Learning Era: A Comprehensive Survey

TL;DR

The paper surveys deep learning-based colorectal polyp segmentation (CPS) from 2014 to 2023, introducing a 26-subcategory taxonomy of network architectures, supervision levels, and learning paradigms. It systematically analyzes 14 CPS datasets, 40 state-of-the-art models, and standard evaluation metrics, while benchmarking performance and assessing out-of-distribution generalization with PolypGen and SUN-SEG attributes. The study highlights key challenges—domain shift, interpretability, privacy, and robustness—and outlines promising directions including federated learning, anomaly localization, lightweight models, and integration with large segmentation and language models. The work aims to provide a standardized, open benchmarking framework to accelerate CPS research and facilitate real-world clinical adoption.

Abstract

Colorectal polyp segmentation (CPS), an essential problem in medical image analysis, has garnered growing research attention. Recently, the deep learning-based model completely overwhelmed traditional methods in the field of CPS, and more and more deep CPS methods have emerged, bringing the CPS into the deep learning era. To help the researchers quickly grasp the main techniques, datasets, evaluation metrics, challenges, and trending of deep CPS, this paper presents a systematic and comprehensive review of deep-learning-based CPS methods from 2014 to 2023, a total of 115 technical papers. In particular, we first provide a comprehensive review of the current deep CPS with a novel taxonomy, including network architectures, level of supervision, and learning paradigm. More specifically, network architectures include eight subcategories, the level of supervision comprises six subcategories, and the learning paradigm encompasses 12 subcategories, totaling 26 subcategories. Then, we provided a comprehensive analysis the characteristics of each dataset, including the number of datasets, annotation types, image resolution, polyp size, contrast values, and polyp location. Following that, we summarized CPS's commonly used evaluation metrics and conducted a detailed analysis of 40 deep SOTA models, including out-of-distribution generalization and attribute-based performance analysis. Finally, we discussed deep learning-based CPS methods' main challenges and opportunities.

Figures (8)

• Figure 1: Colorectal cancers originate from polyps, undergoing a progression from normal mucosa to adenomatous polyps, then pre-cancerous polyps, adenocarcinoma, and finally transforming into cancer. The survival rate of CRC patients exceeds 90% in the first stage but drastically drops to less than 5% in the last stage.
• Figure 2: A brief history of CPS. The first seminal work jerebko2003polyp on CPS can be traced back to 2003. The first deep learning-based CPS method ronneberger2015u emerged in 2015. The methods mentioned above serve as milestones, which are typically highly cited. See Section \ref{['sec: history']} for more details.
• Figure 3: An overview of our proposed taxonomy. We categorize existing polyp segmentation approaches into three branches from different perspectives: network architectures, level of supervision, and learning paradigm. Specifically, network architectures include eight subcategories, the level of supervision comprises six subcategories, and the learning paradigm encompasses 12 subcategories, totaling 26 subcategories.
• Figure 4: Classification of deep CPS models according to the network architectures. (a) MLP-based methods, (b) FCN-based models, (c) UNet-based approaches, (d) Deep supervision network, (e) Transformer-based network, and (f) Multi-branch networks, and (g) Hybrid-network.
• Figure 5: Statistics of polyp size (first two rows) and contrast value (last two rows) of current CPS datasets. From the perspective of polyp size, the BKAI-IGH is the most challenging dataset because it has the smallest average polyp size, while from the perspective of contrast, the SUN-SEG is the most challenging one because its contrast value is only 0.51.