Towards Imbalanced Motion: Part-Decoupling Network for Video Portrait Segmentation
Tianshu Yu, Changqun Xia, Jia Li
TL;DR
This work tackles the challenge of robust video portrait segmentation in complex, multi-scene settings by introducing MVPS, a large-scale dataset with 7 scenario categories and 10,843 pixel-level annotated frames, designed to reveal motion imbalance across portrait parts. To exploit this imbalance, the authors propose Part-Decoupling Network (PDNet) featuring an Inter-frame Part-Discriminated Attention (IPDA) module that unsupervisedly segments portraits into parts, performs cross-part motion correlation via part-specific cross-attention, and assembles the results with spatiotemporal fusion for accurate VPS. The method is trained with a combination of portrait-segmentation losses and self-supervised part-decoupling losses (geo, sem, area), and uses a single reference frame to balance performance and efficiency. Empirically, PDNet achieves state-of-the-art performance among unsupervised VOS methods on MVPS and PP-HumanSeg14K, with strong qualitative results and ablations confirming the benefits of IPDA, the number of parts $p$, and reference-frame settings. This work provides a new benchmark and a principled, part-aware approach for robust VPS in realistic, complex scenes, with potential impact on video editing and AR workflows.
Abstract
Video portrait segmentation (VPS), aiming at segmenting prominent foreground portraits from video frames, has received much attention in recent years. However, simplicity of existing VPS datasets leads to a limitation on extensive research of the task. In this work, we propose a new intricate large-scale Multi-scene Video Portrait Segmentation dataset MVPS consisting of 101 video clips in 7 scenario categories, in which 10,843 sampled frames are finely annotated at pixel level. The dataset has diverse scenes and complicated background environments, which is the most complex dataset in VPS to our best knowledge. Through the observation of a large number of videos with portraits during dataset construction, we find that due to the joint structure of human body, motion of portraits is part-associated, which leads that different parts are relatively independent in motion. That is, motion of different parts of the portraits is imbalanced. Towards this imbalance, an intuitive and reasonable idea is that different motion states in portraits can be better exploited by decoupling the portraits into parts. To achieve this, we propose a Part-Decoupling Network (PDNet) for video portrait segmentation. Specifically, an Inter-frame Part-Discriminated Attention (IPDA) module is proposed which unsupervisedly segments portrait into parts and utilizes different attentiveness on discriminative features specified to each different part. In this way, appropriate attention can be imposed to portrait parts with imbalanced motion to extract part-discriminated correlations, so that the portraits can be segmented more accurately. Experimental results demonstrate that our method achieves leading performance with the comparison to state-of-the-art methods.