CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation

TL;DR

A simple encoder-decoder network, called CLIP-VIS, to adapt CLIP for open-vocabulary video instance segmentation, which adopts frozen CLIP and introduces three modules, including class-agnostic mask generation, temporal topK-enhanced matching, and weighted open-vocabulary classification.

Abstract

Open-vocabulary video instance segmentation strives to segment and track instances belonging to an open set of categories in a videos. The vision-language model Contrastive Language-Image Pre-training (CLIP) has shown robust zero-shot classification ability in image-level open-vocabulary tasks. In this paper, we propose a simple encoder-decoder network, called CLIP-VIS, to adapt CLIP for open-vocabulary video instance segmentation. Our CLIP-VIS adopts frozen CLIP and introduces three modules, including class-agnostic mask generation, temporal topK-enhanced matching, and weighted open-vocabulary classification. Given a set of initial queries, class-agnostic mask generation introduces a pixel decoder and a transformer decoder on CLIP pre-trained image encoder to predict query masks and corresponding object scores and mask IoU scores. Then, temporal topK-enhanced matching performs query matching across frames using the K mostly matched frames. Finally, weighted open-vocabulary classification first employs mask pooling to generate query visual features from CLIP pre-trained image encoder, and second performs weighted classification using object scores and mask IoU scores. Our CLIP-VIS does not require the annotations of instance categories and identities. The experiments are performed on various video instance segmentation datasets, which demonstrate the effectiveness of our proposed method, especially for novel categories. When using ConvNeXt-B as backbone, our CLIP-VIS achieves the AP and APn scores of 32.2% and 40.2% on the validation set of LV-VIS dataset, which outperforms OV2Seg by 11.1% and 23.9% respectively. We will release the source code and models at https://github.com/zwq456/CLIP-VIS.git.

Figures (6)

• Figure 1: Comparisons with existing open-vocabulary video instance segmentation pipelines. The inputs are category names and video frames, and the outputs are masks, categories and matching results across frames of different instances.Left: OV2Seg employs visual features of trainable backbone and CLIP text encoder for open-vocabulary classification. Instance query matching is performed using all previous (long-term) query embeddings. Middle: OpenVIS first utilizes a trainable network to extract proposal masks, and then employs frozen CLIP for classification. Instance query matching is conducted using adjacent query embeddings. Right: Our CLIP-VIS performs mask generation and classification using a single frozen CLIP. Query matching is performed based on query embeddings of $K$ mostly matched frames.
• Figure 2: Overall architecture of our proposed CLIP-VIS. It mainly comprises three modules: a) Class-agnostic mask generation. It employs CLIP image encoder as backbone, and utilizes pixel decoder and transformer decoder in Mask2former to extract query masks. In addition, it generates mask IoU scores and object scores. b) Temporal top$K$-enhanced matching. This module selects $K$ mostly matched frames and associates query embeddings between current frame and these selected frames. c) Weighted open-vocabulary classification. This module extracts query visual features from CLIP backbone using mask pooling, and calculates similarity scores between query visual features and CLIP text embeddings for classification. In addition, we refine the classification by object scores and mask IoU scores.
• Figure 3: Pipeline of temporal top$K$-enhanced matching. We first select $K$ mostly matched frames from last $T$ frames according to frame-level cost values. Afterwards, we perform query matching using selected frames. In addition, we append the query embeddings in current frame in memory bank.
• Figure 4: Visualizing query embeddings and qualitative results. In left column, the points represent the query embeddings of an instance in all frames using t-SNE. Specifically, the circle points with green color represent the query embeddings with correct matching in both adjacent or long-term and top$K$ strategy, the triangle points with blue color represent the query embeddings with correct matching in our top$K$ strategy, and the square points with red color represent the query embeddings with wrong matching in adjacent or long-term strategy. In right column, we show qualitative results corresponding to some points. The first row shows the results of adjacent or long-term strategy, and the second row shows the results of our top$K$ strategy.
• Figure 5: Qualitative results of our proposed method.a) shows some video instance segmentation results on novel categories. b) displays some video instance segmentation results of various bird species, where our method can accurately classify similar categories. c) shows video instance results in complex scenes.