Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation

TL;DR

The paper tackles open-vocabulary video instance segmentation by addressing two core problems: a domain gap between vision-language model features and per-clip instance queries, and underutilization of temporal information. It introduces OVFormer, a lightweight end-to-end framework that performs unified embedding alignment between video-level instance queries and CLIP image embeddings, coupled with video-level training and a semi-online inference scheme to exploit temporal consistency. The approach yields substantial gains over prior methods, achieving 21.9 mAP on LV-VIS with a ResNet-50 backbone and showing strong zero-shot generalization on YouTube-VIS 2019/2021 and OVIS, with Swin-B backbones further boosting performance. These results demonstrate enhanced generalization to novel categories and improved temporal stability, making OVFormer a competitive baseline for open-vocabulary VIS in practical settings.

Abstract

Open-Vocabulary Video Instance Segmentation (VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, the recent Open-Vocabulary VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between the VLM features (e.g., CLIP) and the instance queries and the underutilization of temporal consistency are two central causes. To mitigate these issues, we design and train a novel Open-Vocabulary VIS baseline called OVFormer. OVFormer utilizes a lightweight module for unified embedding alignment between query embeddings and CLIP image embeddings to remedy the domain gap. Unlike previous image-based training methods, we conduct video-based model training and deploy a semi-online inference scheme to fully mine the temporal consistency in the video. Without bells and whistles, OVFormer achieves 21.9 mAP with a ResNet-50 backbone on LV-VIS, exceeding the previous state-of-the-art performance by 7.7. Extensive experiments on some Close-Vocabulary VIS datasets also demonstrate the strong zero-shot generalization ability of OVFormer (+ 7.6 mAP on YouTube-VIS 2019, + 3.9 mAP on OVIS). Code is available at https://github.com/fanghaook/OVFormer.

Figures (6)

• Figure 1: The proposed OVFormer outperforms its counterparts significantly on the VIS datasets.
• Figure 2: Overview of OVFormer. Our proposed OVFormer consists of three modules: Instance Query Generation module generates video-level instance queries that represent the entire clip; Then, Open-Vocabulary Classification module performs unified embedding alignment between the instance queries and the image features of CLIP to generate feature-aligned class embeddings, then classify them with the text embeddings computed from a CLIP text encoder. Finally, Instance Segmentation and Association module generates and tracks the instances through bipartite matching.
• Figure 3: Illustration of different inference schemes. (a) Online inference takes a single frame as input and associates instances frame by frame; (b) Offline inference takes the entire video as input, without associating instances; (c) Semi-online inference takes video clips as input and associates instances clip by clip.
• Figure 4: The performance comparison of base and novel categories on LV-VLIS validation set.
• Figure 5: Ablation study of the clip length $T$ during OVFormer inference on LV-VIS validation set.