Improving Weakly-supervised Video Instance Segmentation by Leveraging Spatio-temporal Consistency

TL;DR

This work introduces a novel weakly-supervised method called Eigen-Cluster VIS that achieves competitive accuracy compared to other VIS approaches, without requiring any mask annotations, and effectively narrows the performance gap between the fully-supervised and weakly-supervised VIS approaches.

Abstract

The performance of Video Instance Segmentation (VIS) methods has improved significantly with the advent of transformer networks. However, these networks often face challenges in training due to the high annotation cost. To address this, unsupervised and weakly-supervised methods have been developed to reduce the dependency on annotations. This work introduces a novel weakly-supervised method called Eigen-Cluster VIS that, without requiring any mask annotations, achieves competitive accuracy compared to other VIS approaches. This method is based on two key innovations: a Temporal Eigenvalue Loss (TEL) and a clip-level Quality Cluster Coefficient (QCC). The TEL ensures temporal coherence by leveraging the eigenvalues of the Laplacian matrix derived from graph adjacency matrices. By minimizing the mean absolute error between the eigenvalues of adjacent frames, this loss function promotes smooth transitions and stable segmentation boundaries over time, reducing temporal discontinuities and improving overall segmentation quality. The QCC employs the K-means method to ensure the quality of spatio-temporal clusters without relying on ground truth masks. Using the Davies-Bouldin score, the QCC provides an unsupervised measure of feature discrimination, allowing the model to self-evaluate and adapt to varying object distributions, enhancing robustness during the testing phase. These enhancements are computationally efficient and straightforward, offering significant performance gains without additional annotated data. The proposed Eigen-Cluster VIS method is evaluated on the YouTube-Video Instance Segmentation (YouTube-VIS) 2019/2021 and Occluded Video Instance Segmentation (OVIS) datasets, demonstrating that it effectively narrows the performance gap between the fully-supervised and weakly-supervised VIS approaches. The code is available on https://github.com/farnooshar/EigenClusterVIS

Figures (6)

• Figure 1: Visualization of Eigen-Cluster VIS Predictions. Eigen-Cluster VIS uses no video or image mask annotations, and achieves 44.2% mask AP on the YouTube-VIS 2019 validation dataset, with a ResNet-50 backbone. These outputs validate the fact that high performance VIS can be obtained without any mask annotations.
• Figure 2: Pipelines of proposed Eigen-Cluster VIS. After inputting $X$ into Mask2former, two outputs, $U$ and $O$, are generated. According to Method ke2022mask, $X$ is also fed into the Temporal Patch Matching module to extract the corresponding patches for TK-Loss. $U$ pertains to classification and is optimized using Cross-Entropy Loss, while $O$ represents instance features from which the QCC, TEL, and TK-Loss modules are derived. All the losses are then aggregated under the title $L_{\text{segmentation}}$ during the training process, and the optimization is carried out.
• Figure 3: Quality Cluster Coefficient. Instance features are reshaped and resized along the spatial dimension for each clip in a batch. Subsequently, a K-means clustering (where K is the number of instances in the ground truth) is performed on these features. The Quality Cluster Coefficient is computed by using the Davies-Bouldin metric.
• Figure 4: Temporal Eigenvalue Loss. Matched instances are identified through sequence matching between the instance features and ground truth masks. Subsequently, for each instance in every frame, the features are resized and reshaped, followed by the creation of an affinity matrix based on these features. Eigenvalues of the Laplacian of this affinity matrix are extracted for each instance. The MAE loss is then computed cyclically between the corresponding instances in every two consecutive frames.
• Figure 5: Visualization of Baseline and Eigen-Cluster VIS Differences. In the baseline, some foreground areas may be missed, particularly in details involving fast movements. These missed regions are indicated with dashed lines.