Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation

Qingfeng Liu, Mostafa El-Khamy, Kee-Bong Song

TL;DR

This work targets Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging wild videos with a three-component DVIS pipeline. It leverages a frozen DINOv2-g backbone augmented by a lightweight ViT-Adapter to align embeddings for the segmenter, online tracker, and offline temporal refiner, achieving state-of-the-art VPS results (VPQ 58.26; STQ 0.543) and strong VSS performance (mIoU ≈0.64, VC16 ≈0.933). Pretraining on VIPSeg followed by finetuning on VSPW yields the best temporal consistency and per-frame accuracy, highlighting the trade-off between tracking stability and boundary precision. The approach demonstrates that sharing a large foundation model across VPS and VSS tasks via an efficient adapter can reduce training costs while delivering top-tier performance and scalable inference for multi-task video understanding.

Abstract

The third Pixel-level Video Understanding in the Wild (PVUW CVPR 2024) challenge aims to advance the state of art in video understanding through benchmarking Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging videos and scenes introduced in the large-scale Video Panoptic Segmentation in the Wild (VIPSeg) test set and the large-scale Video Scene Parsing in the Wild (VSPW) test set, respectively. This paper details our research work that achieved the 1st place winner in the PVUW'24 VPS challenge, establishing state of art results in all metrics, including the Video Panoptic Quality (VPQ) and Segmentation and Tracking Quality (STQ). With minor fine-tuning our approach also achieved the 3rd place in the PVUW'24 VSS challenge ranked by the mIoU (mean intersection over union) metric and the first place ranked by the VC16 (16-frame video consistency) metric. Our winning solution stands on the shoulders of giant foundational vision transformer model (DINOv2 ViT-g) and proven multi-stage Decoupled Video Instance Segmentation (DVIS) frameworks for video understanding.

1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation

TL;DR

This work targets Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging wild videos with a three-component DVIS pipeline. It leverages a frozen DINOv2-g backbone augmented by a lightweight ViT-Adapter to align embeddings for the segmenter, online tracker, and offline temporal refiner, achieving state-of-the-art VPS results (VPQ 58.26; STQ 0.543) and strong VSS performance (mIoU ≈0.64, VC16 ≈0.933). Pretraining on VIPSeg followed by finetuning on VSPW yields the best temporal consistency and per-frame accuracy, highlighting the trade-off between tracking stability and boundary precision. The approach demonstrates that sharing a large foundation model across VPS and VSS tasks via an efficient adapter can reduce training costs while delivering top-tier performance and scalable inference for multi-task video understanding.

Abstract

The third Pixel-level Video Understanding in the Wild (PVUW CVPR 2024) challenge aims to advance the state of art in video understanding through benchmarking Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging videos and scenes introduced in the large-scale Video Panoptic Segmentation in the Wild (VIPSeg) test set and the large-scale Video Scene Parsing in the Wild (VSPW) test set, respectively. This paper details our research work that achieved the 1st place winner in the PVUW'24 VPS challenge, establishing state of art results in all metrics, including the Video Panoptic Quality (VPQ) and Segmentation and Tracking Quality (STQ). With minor fine-tuning our approach also achieved the 3rd place in the PVUW'24 VSS challenge ranked by the mIoU (mean intersection over union) metric and the first place ranked by the VC16 (16-frame video consistency) metric. Our winning solution stands on the shoulders of giant foundational vision transformer model (DINOv2 ViT-g) and proven multi-stage Decoupled Video Instance Segmentation (DVIS) frameworks for video understanding.
Paper Structure (12 sections, 1 figure, 8 tables)

This paper contains 12 sections, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Large Visual Foundation Models
  4. ViT-Adapter design
  5. Finetune for Video Semantic Segmentation
  6. Experiment
  7. Dataset
  8. Implementation Details
  9. Results for Video Panoptic Segmentation Track
  10. Results for Video Semantic Segmentation Track
  11. Ablation Studies
  12. Discussions and Conclusions

Figures (1)

  • Figure 1: The System architecture overview. It consists of three independent components: a segmenter, an online tracker, and an offline temporal refiner. We mainly focus on improve the performance of segmenter by applying ViT-Adapter with ViT-g pretrained by DINOv2.