Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mask2Former for Video Instance Segmentation

Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, Alexander G. Schwing

TL;DR

Mask2Former extends a universal image segmentation model to video by treating a video as a 3D spatio-temporal volume and applying masked attention, a separate temporal positional encoding, and 3D mask prediction. This yields state-of-the-art video instance segmentation on YouTubeVIS-2019 and YouTubeVIS-2021 without modifying the architecture, losses, or training pipeline. The results demonstrate that universal image segmentation frameworks can generalize to video tasks with simple, principled adaptations, potentially enabling broader video semantic and panoptic segmentation research. The work highlights practical, scalable approaches for unified image-video segmentation across domains.

Abstract

We find Mask2Former also achieves state-of-the-art performance on video instance segmentation without modifying the architecture, the loss or even the training pipeline. In this report, we show universal image segmentation architectures trivially generalize to video segmentation by directly predicting 3D segmentation volumes. Specifically, Mask2Former sets a new state-of-the-art of 60.4 AP on YouTubeVIS-2019 and 52.6 AP on YouTubeVIS-2021. We believe Mask2Former is also capable of handling video semantic and panoptic segmentation, given its versatility in image segmentation. We hope this will make state-of-the-art video segmentation research more accessible and bring more attention to designing universal image and video segmentation architectures.

Mask2Former for Video Instance Segmentation

TL;DR

Mask2Former extends a universal image segmentation model to video by treating a video as a 3D spatio-temporal volume and applying masked attention, a separate temporal positional encoding, and 3D mask prediction. This yields state-of-the-art video instance segmentation on YouTubeVIS-2019 and YouTubeVIS-2021 without modifying the architecture, losses, or training pipeline. The results demonstrate that universal image segmentation frameworks can generalize to video tasks with simple, principled adaptations, potentially enabling broader video semantic and panoptic segmentation research. The work highlights practical, scalable approaches for unified image-video segmentation across domains.

Abstract

We find Mask2Former also achieves state-of-the-art performance on video instance segmentation without modifying the architecture, the loss or even the training pipeline. In this report, we show universal image segmentation architectures trivially generalize to video segmentation by directly predicting 3D segmentation volumes. Specifically, Mask2Former sets a new state-of-the-art of 60.4 AP on YouTubeVIS-2019 and 52.6 AP on YouTubeVIS-2021. We believe Mask2Former is also capable of handling video semantic and panoptic segmentation, given its versatility in image segmentation. We hope this will make state-of-the-art video segmentation research more accessible and bring more attention to designing universal image and video segmentation architectures.
Paper Structure (11 sections, 4 equations, 1 figure, 2 tables)

This paper contains 11 sections, 4 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Mask2Former for Videos
  4. Joint spatio-temporal masked attention
  5. Temporal positional encoding
  6. Joint spatio-temporal mask prediction
  7. Experiments
  8. Training
  9. Inference
  10. Results
  11. Acknowledgments:

Figures (1)

  • Figure 1: Mask2Former trivially generalizes to videos. For single-frame input data, it operates as a standard image segmentation architecture. For $>1$ frames, due to sharing of the queries across frames, it segments and tracks object instances across frames.