Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAST: Video Polyp Segmentation with a Mixture-Attention Siamese Transformer

Geng Chen, Junqing Yang, Xiaozhou Pu, Ge-Peng Ji, Huan Xiong, Yongsheng Pan, Hengfei Cui, Yong Xia

TL;DR

MAST tackles video polyp segmentation by jointly encoding paired frames with a Siamese transformer and enriching their representations through a novel mixture-attention mechanism that combines inter-frame mutual-attention with intra-frame self-attention. The architecture uses two parallel decoders to generate segmentation maps and is trained with a hybrid BCE-IoU loss across multi-level outputs. On the SUN-SEG benchmark, MAST delivers state-of-the-art performance while maintaining competitive computation and parameter efficiency, with ablations confirming the effectiveness of both the Siamese transformer and mixture-attention modules. The method’s ability to leverage long-range spatio-temporal cues improves segmentation accuracy for challenging polyps and supports more reliable VPS in clinical workflows.

Abstract

Accurate segmentation of polyps from colonoscopy videos is of great significance to polyp treatment and early prevention of colorectal cancer. However, it is challenging due to the difficulties associated with modelling long-range spatio-temporal relationships within a colonoscopy video. In this paper, we address this challenging task with a novel Mixture-Attention Siamese Transformer (MAST), which explicitly models the long-range spatio-temporal relationships with a mixture-attention mechanism for accurate polyp segmentation. Specifically, we first construct a Siamese transformer architecture to jointly encode paired video frames for their feature representations. We then design a mixture-attention module to exploit the intra-frame and inter-frame correlations, enhancing the features with rich spatio-temporal relationships. Finally, the enhanced features are fed to two parallel decoders for predicting the segmentation maps. To the best of our knowledge, our MAST is the first transformer model dedicated to video polyp segmentation. Extensive experiments on the large-scale SUN-SEG benchmark demonstrate the superior performance of MAST in comparison with the cutting-edge competitors. Our code is publicly available at https://github.com/Junqing-Yang/MAST.

MAST: Video Polyp Segmentation with a Mixture-Attention Siamese Transformer

TL;DR

MAST tackles video polyp segmentation by jointly encoding paired frames with a Siamese transformer and enriching their representations through a novel mixture-attention mechanism that combines inter-frame mutual-attention with intra-frame self-attention. The architecture uses two parallel decoders to generate segmentation maps and is trained with a hybrid BCE-IoU loss across multi-level outputs. On the SUN-SEG benchmark, MAST delivers state-of-the-art performance while maintaining competitive computation and parameter efficiency, with ablations confirming the effectiveness of both the Siamese transformer and mixture-attention modules. The method’s ability to leverage long-range spatio-temporal cues improves segmentation accuracy for challenging polyps and supports more reliable VPS in clinical workflows.

Abstract

Accurate segmentation of polyps from colonoscopy videos is of great significance to polyp treatment and early prevention of colorectal cancer. However, it is challenging due to the difficulties associated with modelling long-range spatio-temporal relationships within a colonoscopy video. In this paper, we address this challenging task with a novel Mixture-Attention Siamese Transformer (MAST), which explicitly models the long-range spatio-temporal relationships with a mixture-attention mechanism for accurate polyp segmentation. Specifically, we first construct a Siamese transformer architecture to jointly encode paired video frames for their feature representations. We then design a mixture-attention module to exploit the intra-frame and inter-frame correlations, enhancing the features with rich spatio-temporal relationships. Finally, the enhanced features are fed to two parallel decoders for predicting the segmentation maps. To the best of our knowledge, our MAST is the first transformer model dedicated to video polyp segmentation. Extensive experiments on the large-scale SUN-SEG benchmark demonstrate the superior performance of MAST in comparison with the cutting-edge competitors. Our code is publicly available at https://github.com/Junqing-Yang/MAST.
Paper Structure (26 sections, 18 equations, 3 figures, 5 tables, 1 algorithm)

This paper contains 26 sections, 18 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Polyp Segmentation
  4. Transformer in Vision
  5. Attention Mechanisms.
  6. Method
  7. Siamese Transformer
  8. Mixture-Attention Module
  9. Feature Embedding
  10. Attention Matrix Calculation
  11. Enhancement and Fusion
  12. Parallel Decoders
  13. Loss Function
  14. Experiments
  15. Implementation Details
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Overview of MAST: Using an anchor frame at $t=T$ and a reference frame at $t=T-\Delta$ as input, MAST employs a Siamese transformer for feature extraction and a Mixture-Attention module to compute the attention matrix to enhance the features. The enhanced features are then fed into two parallel decoders for segmentation map prediction.
  • Figure 2: Comparison of Precision-Recall, F-measure, and E-measure curves between cutting-edge competitors and our MAST on the SUN-SEG-Easy (1$^{st}$ row) and SUN-SEG-Hard datasets (2$^{nd}$ row).
  • Figure 3: The qualitative comparison of cutting-edge competitors and our MAST.