Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Polyp Segmentation Generalisability of Pretrained Backbones

Edward Sanderson, Bogdan J. Matuszewski

TL;DR

This work examines how pretrained backbone choices for polyp segmentation generalise when transferred to a dataset different from the fine-tuning data. By evaluating 12 models with ResNet50 or ViT-B backbones, across diverse pretraining pipelines, on Kvasir-SEG fine-tuning and CVC-ClinicDB testing, it reveals robust generalisation patterns. The key finding is that ImageNet-1k pretraining, especially via self-supervised methods, often yields the strongest cross-domain generalisation, while ResNet50 backbones tend to generalise more reliably than ViT-B backbones in cross-dataset scenarios. These results highlight the nuanced interplay between architecture and pretraining in enabling robust deployment for automated polyp segmentation across varying clinical data distributions, informing backbone selection for real-world colonoscopy analysis.

Abstract

It has recently been demonstrated that pretraining backbones in a self-supervised manner generally provides better fine-tuned polyp segmentation performance, and that models with ViT-B backbones typically perform better than models with ResNet50 backbones. In this paper, we extend this recent work to consider generalisability. I.e., we assess the performance of models on a different dataset to that used for fine-tuning, accounting for variation in network architecture and pretraining pipeline (algorithm and dataset). This reveals how well models with different pretrained backbones generalise to data of a somewhat different distribution to the training data, which will likely arise in deployment due to different cameras and demographics of patients, amongst other factors. We observe that the previous findings, regarding pretraining pipelines for polyp segmentation, hold true when considering generalisability. However, our results imply that models with ResNet50 backbones typically generalise better, despite being outperformed by models with ViT-B backbones in evaluation on the test set from the same dataset used for fine-tuning.

Polyp Segmentation Generalisability of Pretrained Backbones

TL;DR

This work examines how pretrained backbone choices for polyp segmentation generalise when transferred to a dataset different from the fine-tuning data. By evaluating 12 models with ResNet50 or ViT-B backbones, across diverse pretraining pipelines, on Kvasir-SEG fine-tuning and CVC-ClinicDB testing, it reveals robust generalisation patterns. The key finding is that ImageNet-1k pretraining, especially via self-supervised methods, often yields the strongest cross-domain generalisation, while ResNet50 backbones tend to generalise more reliably than ViT-B backbones in cross-dataset scenarios. These results highlight the nuanced interplay between architecture and pretraining in enabling robust deployment for automated polyp segmentation across varying clinical data distributions, informing backbone selection for real-world colonoscopy analysis.

Abstract

It has recently been demonstrated that pretraining backbones in a self-supervised manner generally provides better fine-tuned polyp segmentation performance, and that models with ViT-B backbones typically perform better than models with ResNet50 backbones. In this paper, we extend this recent work to consider generalisability. I.e., we assess the performance of models on a different dataset to that used for fine-tuning, accounting for variation in network architecture and pretraining pipeline (algorithm and dataset). This reveals how well models with different pretrained backbones generalise to data of a somewhat different distribution to the training data, which will likely arise in deployment due to different cameras and demographics of patients, amongst other factors. We observe that the previous findings, regarding pretraining pipelines for polyp segmentation, hold true when considering generalisability. However, our results imply that models with ResNet50 backbones typically generalise better, despite being outperformed by models with ViT-B backbones in evaluation on the test set from the same dataset used for fine-tuning.
Paper Structure (3 sections, 2 figures, 1 table)

This paper contains 3 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Analysis
  3. Conclusion

Figures (2)

  • Figure 1: Comparison of the distribution of instance-wise Dice score from each model's evaluation on the Kvasir-SEG test set (blue) against the distribution from its evaluation on CVC-ClinicDB (red). For conciseness, we denote ResNet50s with RN, ViT-Bs with VT, Hyperkvasir-unlabelled with HK, ImageNet-1k with IN, MoCo v3 with MC, Barlow Twins with BT, MAE with MA, supervised pretraining with SL, and no pretraining with NA-NA.
  • Figure 2: Relative drop in mDice from each model's evaluation on the Kvasir-SEG test set to its evaluation on CVC-ClinicDB. For conciseness, we denote ResNet50s with RN, ViT-Bs with VT, Hyperkvasir-unlabelled with HK, ImageNet-1k with IN, MoCo v3 with MC, Barlow Twins with BT, MAE with MA, supervised pretraining with SL, and no pretraining with NA-NA. For clarity, the results for ResNet50 models are coloured blue and the results for ViT-B models are coloured red.