Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning

Yingling Lu, Yijun Yang, Zhaohu Xing, Qiong Wang, Lei Zhu

TL;DR

This paper incorporates multi-task supervision into diffusion models to promote the discrimination of diffusion models on pixel-by-pixel segmentation and equip TRM with a generative adversarial self-supervised strategy to produce more realistic frames and thus capture better dynamic cues.

Abstract

Diffusion Probabilistic Models have recently attracted significant attention in the community of computer vision due to their outstanding performance. However, while a substantial amount of diffusion-based research has focused on generative tasks, no work introduces diffusion models to advance the results of polyp segmentation in videos, which is frequently challenged by polyps' high camouflage and redundant temporal cues.In this paper, we present a novel diffusion-based network for video polyp segmentation task, dubbed as Diff-VPS. We incorporate multi-task supervision into diffusion models to promote the discrimination of diffusion models on pixel-by-pixel segmentation. This integrates the contextual high-level information achieved by the joint classification and detection tasks. To explore the temporal dependency, Temporal Reasoning Module (TRM) is devised via reasoning and reconstructing the target frame from the previous frames. We further equip TRM with a generative adversarial self-supervised strategy to produce more realistic frames and thus capture better dynamic cues. Extensive experiments are conducted on SUN-SEG, and the results indicate that our proposed Diff-VPS significantly achieves state-of-the-art performance. Code is available at https://github.com/lydia-yllu/Diff-VPS.

Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning

TL;DR

This paper incorporates multi-task supervision into diffusion models to promote the discrimination of diffusion models on pixel-by-pixel segmentation and equip TRM with a generative adversarial self-supervised strategy to produce more realistic frames and thus capture better dynamic cues.

Abstract

Diffusion Probabilistic Models have recently attracted significant attention in the community of computer vision due to their outstanding performance. However, while a substantial amount of diffusion-based research has focused on generative tasks, no work introduces diffusion models to advance the results of polyp segmentation in videos, which is frequently challenged by polyps' high camouflage and redundant temporal cues.In this paper, we present a novel diffusion-based network for video polyp segmentation task, dubbed as Diff-VPS. We incorporate multi-task supervision into diffusion models to promote the discrimination of diffusion models on pixel-by-pixel segmentation. This integrates the contextual high-level information achieved by the joint classification and detection tasks. To explore the temporal dependency, Temporal Reasoning Module (TRM) is devised via reasoning and reconstructing the target frame from the previous frames. We further equip TRM with a generative adversarial self-supervised strategy to produce more realistic frames and thus capture better dynamic cues. Extensive experiments are conducted on SUN-SEG, and the results indicate that our proposed Diff-VPS significantly achieves state-of-the-art performance. Code is available at https://github.com/lydia-yllu/Diff-VPS.
Paper Structure (9 sections, 6 equations, 2 figures, 3 tables)

This paper contains 9 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Multi-task Diffusion Model (MDM)
  4. Temporal Reasoning Module (TRM)
  5. Experiment
  6. Datasets and Experimental Setting
  7. Comparison with State-of-the-Arts
  8. Ablation Study
  9. Conclusion

Figures (2)

  • Figure 1: Overview of our Diff-VPS framework. While the TRM extracts multi-scale temporal features $\mathcal{R}_j$ from the previous frames, the image encoder learns the spatial counterpart $\mathcal{S}_j$. The spatiotemporal prior $\mathcal{H}_j$ from $\mathcal{R}_j$ and $\mathcal{S}_j$ conditions the denoising process of our multi-task diffusion model.
  • Figure 2: Qualitative results on SUN-SEG. Left: a clip from easy-seen dataset case 75. Right: a clip from hard-unseen dataset case 36.