FlowCut: Unsupervised Video Instance Segmentation via Temporal Mask Matching

TL;DR

FlowCut tackles unlabeled video instance segmentation by building pseudo-labels from an affinity matrix $W$ constructed as a convex combination of RGB-feature similarity and optical-flow-feature similarity, thresholded at $\tau$; pseudo-masks are refined iteratively, and two-frame segments are formed by IoU matching with a threshold of $0.5$. A VideoMask2Former model is trained on the curated data, using YouTubeVIS-2021 as the primary source and optional ImageNet-derived pseudo-labels for DAVIS tasks. The method achieves state-of-the-art results on YouTubeVIS-2019/2021 and DAVIS-2017 benchmarks, demonstrating that unsupervised pseudo-labels can rival supervised performance while substantially reducing labeling costs. FlowCut thus provides a scalable, resource-efficient approach to video instance segmentation with strong practical impact.

Abstract

We propose FlowCut, a simple and capable method for unsupervised video instance segmentation consisting of a three-stage framework to construct a high-quality video dataset with pseudo labels. To our knowledge, our work is the first attempt to curate a video dataset with pseudo-labels for unsupervised video instance segmentation. In the first stage, we generate pseudo-instance masks by exploiting the affinities of features from both images and optical flows. In the second stage, we construct short video segments containing high-quality, consistent pseudo-instance masks by temporally matching them across the frames. In the third stage, we use the YouTubeVIS-2021 video dataset to extract our training instance segmentation set, and then train a video segmentation model. FlowCut achieves state-of-the-art performance on the YouTubeVIS-2019, YouTubeVIS-2021, DAVIS-2017, and DAVIS-2017 Motion benchmarks.

Figures (4)

• Figure 1: Qualitative comparison of VideoCutler wang2024videocutler and our method FlowCut on 3 videos from DAVIS-2017 pont20172017 validation set. In the first video, we can accurately detect/track both the bicycle and the cycler. In the second video, we can estimate the moving car's boundary more accurately thanks to the optical flow signal guiding our training. In the third video, we can distinguish different people dancing in the hall better.
• Figure 2: Illustration of the proposed synthetic image pair and corresponding pseudo-masks construction method. Top: We extract the pseudo-masks from each separate frame of a video. Bottom: We match the pseudo-masks across two frames (up to 4 time steps apart).
• Figure 3: Illustration of the proposed mask-matching algorithm. Top row: If the mask indexing across the two frames is inconsistent, the IoUs can be used to rectify the correspondences. Bottom row: If masks are missing in one of the 2 frames, they are considered unreliable, and the IoUs can also be used to remove them.
• Figure 4: Some example cases of the failures from YouTubeVIS-2021 vis2021. The first row shows the input images, the second row shows the pseudo-masks, and the third row shows the ground-truth masks.