SyncVIS: Synchronized Video Instance Segmentation

TL;DR

SyncVIS targets the core challenge of video instance segmentation by adopting a synchronized modeling paradigm that unifies frame- and video-level representations within a DETR-based framework. It introduces explicit video-level embeddings and two modules: (i) synchronized video-frame modeling for mutual refinement of frame- and video-level queries, and (ii) synchronized embedding optimization that divides videos into clips to ease bipartite matching. Empirically, SyncVIS achieves state-of-the-art performance on YouTube-VIS 2019/2021/2022 and OVIS across online and offline settings, outperforming prior methods by consistent margins across backbones and long-video scenarios. The work demonstrates that synchronized representations and clip-based optimization can effectively model complex motion and occlusion, providing a generally applicable enhancement to DETR-based VIS pipelines with practical implications for real-world video analysis tasks.

Abstract

Recent DETR-based methods have advanced the development of Video Instance Segmentation (VIS) through transformers' efficiency and capability in modeling spatial and temporal information. Despite harvesting remarkable progress, existing works follow asynchronous designs, which model video sequences via either video-level queries only or adopting query-sensitive cascade structures, resulting in difficulties when handling complex and challenging video scenarios. In this work, we analyze the cause of this phenomenon and the limitations of the current solutions, and propose to conduct synchronized modeling via a new framework named SyncVIS. Specifically, SyncVIS explicitly introduces video-level query embeddings and designs two key modules to synchronize video-level query with frame-level query embeddings: a synchronized video-frame modeling paradigm and a synchronized embedding optimization strategy. The former attempts to promote the mutual learning of frame- and video-level embeddings with each other and the latter divides large video sequences into small clips for easier optimization. Extensive experimental evaluations are conducted on the challenging YouTube-VIS 2019 & 2021 & 2022, and OVIS benchmarks and SyncVIS achieves state-of-the-art results, which demonstrates the effectiveness and generality of the proposed approach. The code is available at https://github.com/rkzheng99/SyncVIS.

Figures (9)

• Figure 1: Comparison of video instance segmentation paradigms. Previous methods (left part) like VITA heo2022vita adopt asynchronous query-sensitive structures to model instance appearances and trajectories. Our model (right part) employs frame and video embeddings in a query-robust synchronous manner, and they synchronize with each other through the transformer decoder to generate the refined video-level query embeddings for the prediction. Also, we employ a synchronized embedding optimization strategy 'Sync. Optim.' instead of the classic optimization approach.
• Figure 2: Overview of the proposed synchronous video-frame modeling framework SyncVIS. The developed synchronized video-frame modeling paradigm enables video-level embeddings and frame-level ones to synchronize with each other in each stage of the decoder. SyncVIS also suggests a new synchronized embedding optimization strategy. As shown in the right part, SyncVIS decouples the input video frames into several sub-clips and feeds each sub-clip into the mask and classification head. By applying these modules, SyncVIS can incorporate both semantics and movement of instances in each frame in a synchronous manner for superior characterizing ability.
• Figure 3: Ablation study on the complexity of video scenarios regarding the number of input frames $T$.
• Figure 4: Results comparison on the YouTube-VIS 2022 long videos.
• Figure 4: Visual comparison of our SyncVIS with Mask2Former-VIS ('M2F') cheng2021mask2former and VITA heo2022vita. SyncVIS shows impressive accuracy in long, complex scenarios where objects share similar appearances and have heavy occlusions.