Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalize Polyp Segmentation via Inpainting across Diverse Backgrounds and Pseudo-Mask Refinement

Jiajian Ma, Fangqi Lu, Silin Huang, Song Wu, Zhen Li

TL;DR

The paper tackles the domain gap in polyp segmentation by generating diverse, realistic polyp images through diffusion-based inpainting and using the resulting region-guided masks to refine pseudo-labels. It couples a ControlNet-enhanced inpainting model with a pseudo-mask refinement network and a principled sample-selection scheme that favors well-aligned and challenging synthetic cases. Experimental results show that this augmentation improves generalization to external datasets, sometimes matching or surpassing fully supervised baselines, and ablations confirm the additive value of pseudo-mask refinement and selective sampling. The approach leverages readily available negative endoscopy backgrounds to mitigate data privacy concerns and improve real-world applicability of polyp segmentation systems.

Abstract

Inpainting lesions within different normal backgrounds is a potential method of addressing the generalization problem, which is crucial for polyp segmentation models. However, seamlessly introducing polyps into complex endoscopic environments while simultaneously generating accurate pseudo-masks remains a challenge for current inpainting methods. To address these issues, we first leverage the pre-trained Stable Diffusion Inpaint and ControlNet, to introduce a robust generative model capable of inpainting polyps across different backgrounds. Secondly, we utilize the prior that synthetic polyps are confined to the inpainted region, to establish an inpainted region-guided pseudo-mask refinement network. We also propose a sample selection strategy that prioritizes well-aligned and hard synthetic cases for further model fine-tuning. Experiments demonstrate that our inpainting model outperformed baseline methods both qualitatively and quantitatively in inpainting quality. Moreover, our data augmentation strategy significantly enhances the performance of polyp segmentation models on external datasets, achieving or surpassing the level of fully supervised training benchmarks in that domain. Our code is available at https://github.com/497662892/PolypInpainter.

Generalize Polyp Segmentation via Inpainting across Diverse Backgrounds and Pseudo-Mask Refinement

TL;DR

The paper tackles the domain gap in polyp segmentation by generating diverse, realistic polyp images through diffusion-based inpainting and using the resulting region-guided masks to refine pseudo-labels. It couples a ControlNet-enhanced inpainting model with a pseudo-mask refinement network and a principled sample-selection scheme that favors well-aligned and challenging synthetic cases. Experimental results show that this augmentation improves generalization to external datasets, sometimes matching or surpassing fully supervised baselines, and ablations confirm the additive value of pseudo-mask refinement and selective sampling. The approach leverages readily available negative endoscopy backgrounds to mitigate data privacy concerns and improve real-world applicability of polyp segmentation systems.

Abstract

Inpainting lesions within different normal backgrounds is a potential method of addressing the generalization problem, which is crucial for polyp segmentation models. However, seamlessly introducing polyps into complex endoscopic environments while simultaneously generating accurate pseudo-masks remains a challenge for current inpainting methods. To address these issues, we first leverage the pre-trained Stable Diffusion Inpaint and ControlNet, to introduce a robust generative model capable of inpainting polyps across different backgrounds. Secondly, we utilize the prior that synthetic polyps are confined to the inpainted region, to establish an inpainted region-guided pseudo-mask refinement network. We also propose a sample selection strategy that prioritizes well-aligned and hard synthetic cases for further model fine-tuning. Experiments demonstrate that our inpainting model outperformed baseline methods both qualitatively and quantitatively in inpainting quality. Moreover, our data augmentation strategy significantly enhances the performance of polyp segmentation models on external datasets, achieving or surpassing the level of fully supervised training benchmarks in that domain. Our code is available at https://github.com/497662892/PolypInpainter.
Paper Structure (14 sections, 3 figures, 4 tables)

This paper contains 14 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. Diffusion-based polyp inpainting model
  4. Pseudo-mask refinement network
  5. Sample selection strategy
  6. EXPERIMENTS AND DISCUSSION
  7. Datasets
  8. Implementation details
  9. Evaluation of polyp inpainting quality
  10. Generalization in polyp segmentation
  11. Ablation study
  12. CONCLUSION
  13. COMPLIANCE WITH ETHICAL STANDARDS
  14. Acknowledgments

Figures (3)

  • Figure 1: Example of inpainting in external background and pseudo-mask refinement.
  • Figure 2: The framework of our proposed data augmentation method.
  • Figure 3: Qualitative results of polyp inpainting methods. BG: background, Ref: reference, SD: Stable Diffusion Inpaint, Poisson: Poisson image blending.