Unsupervised Pre-training with Language-Vision Prompts for Low-Data Instance Segmentation

TL;DR

A new method, Unsupervised Pre-training with Language-Vision Prompts (UPLVP), which improves QEIS models’ instance segmentation by bringing language-vision prompts to queries/kernels, and can converge faster and perform better than CNN-based models in low-data regimes.

Abstract

In recent times, following the paradigm of DETR (DEtection TRansformer), query-based end-to-end instance segmentation (QEIS) methods have exhibited superior performance compared to CNN-based models, particularly when trained on large-scale datasets. Nevertheless, the effectiveness of these QEIS methods diminishes significantly when confronted with limited training data. This limitation arises from their reliance on substantial data volumes to effectively train the pivotal queries/kernels that are essential for acquiring localization and shape priors. To address this problem, we propose a novel method for unsupervised pre-training in low-data regimes. Inspired by the recently successful prompting technique, we introduce a new method, Unsupervised Pre-training with Language-Vision Prompts (UPLVP), which improves QEIS models' instance segmentation by bringing language-vision prompts to queries/kernels. Our method consists of three parts: (1) Masks Proposal: Utilizes language-vision models to generate pseudo masks based on unlabeled images. (2) Prompt-Kernel Matching: Converts pseudo masks into prompts and injects the best-matched localization and shape features to their corresponding kernels. (3) Kernel Supervision: Formulates supervision for pre-training at the kernel level to ensure robust learning. With the help of our pre-training method, QEIS models can converge faster and perform better than CNN-based models in low-data regimes. Experimental evaluations conducted on MS COCO, Cityscapes, and CTW1500 datasets indicate that the QEIS models' performance can be significantly improved when pre-trained with our method. Code will be available at: https://github.com/lifuguan/UPLVP.

Figures (8)

• Figure 1: Performance of Mask2former in different methods using Mask R-CNN as the benchmark. Here ConvNeXt-V2 is used as the backbone of Mask R-CNN. Mask2former is able to outperform Mask R-CNN with large-scale datasets (COCO-full), but it cannot perform as well as Mask R-CNN with small datasets (COCO-10%, Cityscapes and CTW1500) since it's hard to learn localization and shape priors. Our proposed unsupervised pre-training method based on language-vision prompts not only boosts the vanilla Mask2former significantly but also helps to achieve comparable performance compared with Mask R-CNN.
• Figure 2: Pipeline of our proposed unsupervised pre-training framework UPLVP. It mainly contains three modules: proposal generation, prompt kernel matching and auxiliary supervision. Here we leverage CLIP and OpenSeg to extract text and vision features respectively. As can be seen, our method is parameter-free. Parts in orange color denote our pre-training method with the corresponding supervision. Parts in Gray color denote a vanilla QEIS model, where we take K-Net as an example which contains backbone and kernel update iteration head.
• Figure 3: AP learning curves of K-Net with different pre-training methods on COCO with 10% annotated images. As can be seen, our method can speed up the convergence during fin-tuning phase.
• Figure 4: AP learning curves of Mask-RCNN, vanilla K-Net and K-Net pre-trained with different methods. COCO train2017 is used for pre-training, Cityscapes and CTW1500 are used for fine-tuning for all methods.
• Figure 5: Pipeline of the few-shot instance segmentation task. The query image and the support images are fed into the shared backbone and the extracted support features are injected into all kernels. In this experiment, we adopt COCO2VOC setting where we train models on the subset of COCO train2017 and test them on VOC2012.