OneVOS: Unifying Video Object Segmentation with All-in-One Transformer Framework

TL;DR

OneVOS addresses the fragmentation of traditional semi-supervised video object segmentation by unifying feature extraction, matching, memory management, and multi-object aggregation within an All-in-One Transformer. It introduces Mask Embedding, Token Memory, and Mask Decoding, supported by Unidirectional Hybrid Attention to prevent semantic ambiguity and a Dynamic Token Selector for efficiency. The approach achieves state-of-the-art results across seven datasets, with notable gains on long and complex sequences such as LVOS and MOSE, and demonstrates robust performance when adapted to different backbones and data regimes. The work highlights a shift toward end-to-end, globally optimized VOS, offering practical improvements in speed and accuracy and a clear path for future exploration and reproducibility.

Abstract

Contemporary Video Object Segmentation (VOS) approaches typically consist stages of feature extraction, matching, memory management, and multiple objects aggregation. Recent advanced models either employ a discrete modeling for these components in a sequential manner, or optimize a combined pipeline through substructure aggregation. However, these existing explicit staged approaches prevent the VOS framework from being optimized as a unified whole, leading to the limited capacity and suboptimal performance in tackling complex videos. In this paper, we propose OneVOS, a novel framework that unifies the core components of VOS with All-in-One Transformer. Specifically, to unify all aforementioned modules into a vision transformer, we model all the features of frames, masks and memory for multiple objects as transformer tokens, and integrally accomplish feature extraction, matching and memory management of multiple objects through the flexible attention mechanism. Furthermore, a Unidirectional Hybrid Attention is proposed through a double decoupling of the original attention operation, to rectify semantic errors and ambiguities of stored tokens in OneVOS framework. Finally, to alleviate the storage burden and expedite inference, we propose the Dynamic Token Selector, which unveils the working mechanism of OneVOS and naturally leads to a more efficient version of OneVOS. Extensive experiments demonstrate the superiority of OneVOS, achieving state-of-the-art performance across 7 datasets, particularly excelling in complex LVOS and MOSE datasets with 70.1% and 66.4% $J \& F$ scores, surpassing previous state-of-the-art methods by 4.2% and 7.0%, respectively. And our code will be available for reproducibility and further research.

Figures (7)

• Figure 1: Four different modeling for semi-supervised video object segmentation pipelines. (a) Traditional discrete modeling of VOS. (b) Unified multiple object modeling in one reference. (c) Unified single-object feature extraction and matching. (d) OneVOS: Integrates all core modules with All-in-One Transformer.
• Figure 2: (a) Hybrid Attention Block. (b) Blocks with Dynamic Token Selector. (c) Overview of OneVOS framework equipped with All-in-One Transformer. The reference mask is initially encoded into mask embeddings. Patch embeddings are generated from both the reference and current frames, which are then processed by the Transformer. Within each Transformer layer, mask embeddings are integrated with reference tokens. Then they are selected by Dynamic Token Selector and concatenated with tokens stored in memory and current frame feeded into the Hybrid Attention Module. Finally, the features of current frame output from Transformer are subsequently decoded to multi-object masks.
• Figure 3: (a) Visualization of attention distribution of target areas in OneVOS, showing the alternation between self-attention and cross-attention across layers. (b) Analysis of the maximal attention weights reveals a pattern of alternation between feature extraction via self-attention (notably in layers 3 and 5) and matching via cross-attention (as in layers 2 and 8). (c) The proportional analysis of the attention weights' sum, normalized to one, underscores the non-sequential prioritization across different layers between self-attention and cross-attention.
• Figure 4: (a) Visualization of the selected tokens of reference frame by DTS in different Transformer layers. (b) Adaptive selection ratios learned by DTS in different layers without any constraints.
• Figure 5: Visualization dynamic memory updates of OneVOS utilizing DTS. The selected tokens in red boxes are derived output from DTS as Fig. \ref{['fig:dts_selection']}(a).