A Temporal Modeling Framework for Video Pre-Training on Video Instance Segmentation

TL;DR

This paper addresses the mismatch between image-based pre-training and video-based fine-tuning in video instance segmentation (VIS). It introduces a video pre-training framework that combines consistent pseudo-video augmentation with a multi-scale temporal module (MSTM) to capture short- and long-term temporal relations, enabling temporal knowledge transfer without constraining VIS architectures. The approach, including Consistent Stochastic augmentation (CoSt), Video Morph & Splice (VMoSp), and shift-window multi-head attention coupled with ConvGRU, yields state-of-the-art results on YouTubeVIS and OVIS benchmarks, with up to a 4.0 percentage point AP improvement on OVIS. Additionally, video pre-training enhances image instance segmentation performance, indicating broad robustness and cross-task benefits. Overall, the method significantly narrows the gap between image pre-training and video fine-tuning and offers a versatile path to improve VIS across datasets and architectures.

Abstract

Contemporary Video Instance Segmentation (VIS) methods typically adhere to a pre-train then fine-tune regime, where a segmentation model trained on images is fine-tuned on videos. However, the lack of temporal knowledge in the pre-trained model introduces a domain gap which may adversely affect the VIS performance. To effectively bridge this gap, we present a novel video pre-training approach to enhance VIS models, especially for videos with intricate instance relationships. Our crucial innovation focuses on reducing disparities between the pre-training and fine-tuning stages. Specifically, we first introduce consistent pseudo-video augmentations to create diverse pseudo-video samples for pre-training while maintaining the instance consistency across frames. Then, we incorporate a multi-scale temporal module to enhance the model's ability to model temporal relations through self- and cross-attention at short- and long-term temporal spans. Our approach does not set constraints on model architecture and can integrate seamlessly with various VIS methods. Experiment results on commonly adopted VIS benchmarks show that our method consistently outperforms state-of-the-art methods. Our approach achieves a notable 4.0% increase in average precision on the challenging OVIS dataset.

Figures (5)

• Figure 1: Augmentation comparison.
• Figure 2: Overview of VIS model integrated with multi-scale temporal module.
• Figure 3: Visualization of query embeddings posts pre-training with images and videos using T-SNE.
• Figure 4: Qualitative results on YouTubeVIS-2019 vis.
• Figure 5: Failure cases on YouTubeVIS-2019 vis.