General and Task-Oriented Video Segmentation

TL;DR

GvSeg tackles the lack of a universal model for multiple video segmentation tasks by introducing a task-oriented property accommodation framework that disentangles segment targets into appearance, shape, and position cues. It integrates a shape-position descriptor and Shape-Position-Aware query matching into a single architecture, augmented by task-specific query initialization, association, and a task-aware temporal contrastive learning strategy. The approach yields state-of-the-art or leading results across EVS, VIS, VSS, and VPS on seven benchmark datasets, with ablations confirming the value of each component. This work advances practical multi-task video perception by enabling a unified, extensible model with strong generalization across diverse segmentation tasks.

Abstract

We present GvSeg, a general video segmentation framework for addressing four different video segmentation tasks (i.e., instance, semantic, panoptic, and exemplar-guided) while maintaining an identical architectural design. Currently, there is a trend towards developing general video segmentation solutions that can be applied across multiple tasks. This streamlines research endeavors and simplifies deployment. However, such a highly homogenized framework in current design, where each element maintains uniformity, could overlook the inherent diversity among different tasks and lead to suboptimal performance. To tackle this, GvSeg: i) provides a holistic disentanglement and modeling for segment targets, thoroughly examining them from the perspective of appearance, position, and shape, and on this basis, ii) reformulates the query initialization, matching and sampling strategies in alignment with the task-specific requirement. These architecture-agnostic innovations empower GvSeg to effectively address each unique task by accommodating the specific properties that characterize them. Extensive experiments on seven gold-standard benchmark datasets demonstrate that GvSeg surpasses all existing specialized/general solutions by a significant margin on four different video segmentation tasks.

Figures (13)

• Figure 1: (a) We render holistic modeling on segment targets by disentangling them into appearance, shape and position. (b) By adjusting the involvement of the above three factors into tracking and segmentation according to task requirement, GvSeg achieves remarkable improvement compared to prior top-leading general solutions.
• Figure 2: Illustration of shape-position descriptor (§\ref{['sec:3.2']}).
• Figure 3: (a) Task-oriented queries initialization. (b) Task-oriented object association tailored w.r.t.thing and stuff objects. (c) Shape- and position-aware query matching.
• Figure 4: Illustration of task-oriented temporal contrastive learning (§\ref{['sec:3.2']}). Prior work considers solely instance objects, and samples are restricted within neighbor frames. In UvSeg, instance & thing samples are collected from the whole video according to shape and location similarity, while semantic & stuff samples are gathered from the entire training set to capture diver shapes and appearances of each semantic class.
• Figure 5: Visual comparison results on VIPSeg-VPS miao2022large, YouTube-VIS$_{21}$yang2019video, VSPW-VSS miao2021vspw and YouTube-VOS$_{18}$xu2018youtube (§\ref{['sec:qualitative']}).