Scalable Video Object Segmentation with Identification Mechanism

TL;DR

This work tackles scalable multi-object semi-supervised VOS by introducing two Transformer-based frameworks, AOT and AOST. AOT uses an ID-based mechanism to embed multiple objects into a shared identity space for end-to-end multi-object matching and decoding, while LSTT enables hierarchical long- and short-term propagation. AOST further provides online scalability via Scalable LSTT and layer-wise ID-based attention, coupled with scalable supervision to support depth-adaptive inference. The authors validate their approaches on VOSW and standard benchmarks, showing state-of-the-art accuracy and significant speedups, thereby advancing practical multi-object VOS in diverse deployment scenarios.

Abstract

This paper delves into the challenges of achieving scalable and effective multi-object modeling for semi-supervised Video Object Segmentation (VOS). Previous VOS methods decode features with a single positive object, limiting the learning of multi-object representation as they must match and segment each target separately under multi-object scenarios. Additionally, earlier techniques catered to specific application objectives and lacked the flexibility to fulfill different speed-accuracy requirements. To address these problems, we present two innovative approaches, Associating Objects with Transformers (AOT) and Associating Objects with Scalable Transformers (AOST). In pursuing effective multi-object modeling, AOT introduces the IDentification (ID) mechanism to allocate each object a unique identity. This approach enables the network to model the associations among all objects simultaneously, thus facilitating the tracking and segmentation of objects in a single network pass. To address the challenge of inflexible deployment, AOST further integrates scalable long short-term transformers that incorporate scalable supervision and layer-wise ID-based attention. This enables online architecture scalability in VOS for the first time and overcomes ID embeddings' representation limitations. Given the absence of a benchmark for VOS involving densely multi-object annotations, we propose a challenging Video Object Segmentation in the Wild (VOSW) benchmark to validate our approaches. We evaluated various AOT and AOST variants using extensive experiments across VOSW and five commonly used VOS benchmarks, including YouTube-VOS 2018 & 2019 Val, DAVIS-2017 Val & Test, and DAVIS-2016. Our approaches surpass the state-of-the-art competitors and display exceptional efficiency and scalability consistently across all six benchmarks. Project page: https://github.com/yoxu515/aot-benchmark.

Figures (9)

• Figure 1: Post-ensemble methods (e.g., spacetimecfbi) process each object individually in multi-object scenarios (a). Such single-object modeling is ineffective in exploring multi-object contextual information. In contrast, our approaches associate all objects collaboratively (b), which learns more robust multi-object representation, also resulting in better efficiency (c).
• Figure 2: (a) Previous VOS methods (e.g., cfbipNEURIPS2020_liangVOS) are not flexible for different deployment requirements. (b) Our AOST approach supports run-time adjustment for speed-accuracy trade-offs. (c) Under multi-object scenarios, AOST is online scalable and significantly outperforms state-of-the-art competitors cfbicfbipcheng2021stcn on both speed and accuracy. All the results are fairly recorded (using 1 Tesla V100 GPU, full-precision) on YouTube-VOS 2018 youtubevos.
• Figure 3: The overview of AOST (§\ref{['sec:method']}). (a)(b) Inference and training of AOST. The multi-object masks are embedded by using our identification mechanism. Moreover, an S-LSTT with dynamic depth ($n_l'=1/2/3$) is responsible for matching multiple objects collaboratively and hierarchically. (c) An illustration of the IDentity assignment (ID) designed for transferring a $n_{obj}$-object mask into an identification embedding. (d) The structure of an S-LSTT block. LN: layer normalization ln.
• Figure 4: Illustrations of long-term attention and short-term attention (§\ref{['sec:LSTT']}). (a) Long-term attention employs a non-local manner to match all the locations in long-term memory. (b) In contrast, the short-term attention only focuses on a nearby spatial-temporal region with a shape of ${n_{\mathrm{ST}}}\lambda^2$.
• Figure 5: Layer-wise ID-based Attention (§\ref{['sec:id_att']}) is improved with trainable identification weight $W^{id}_{l}$ and gate weight $W^{gate}_{l}$, compared to the vanilla ID-based attention (Eq.\ref{['eq: attid']})