VidEoMT: Your ViT is Secretly Also a Video Segmentation Model

TL;DR

The Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules, and employs a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame.

Abstract

Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/

Figures (19)

• Figure 1: CAVIS vs. VidEoMT (Ours). VidEoMT is much faster than both CAVIS lee2025cavis and a combination of EoMT kerssies2025eomt and CAVIS, while maintaining competitive AP across different sizes of DINOv2 oquab2023dinov2. Evaluated on YouTube-VIS 2019 valyang2019video.
• Figure 2: Current State-of-the-Art Video Segmentation Methods vs. VidEoMT (Ours). We compare the architectures of current state-of-the-art video segmentation methods -- using CAVIS lee2025cavis as a representative example -- and our encoder-only VidEoMT method. VidEoMT streamlines the video segmentation framework, relying on the power of large-scale pre-training with vision foundation models rather than handcrafted task-specific components. TF means Transformer and CA means context-aware.
• Figure 3: VidEoMT architecture. For the initial video frame at $t = 0$, learnable queries are concatenated to the patch tokens after the first $L_1$ ViT blocks. Both sets of tokens are then jointly processed in the last $L_2$ blocks, outputting predictions and track queries. For successive frames, learnable queries and previous-frame track queries are fed to the query fusion module before being processed by the ViT blocks.
• Figure A: Removing specialized components. This figure visualizes the step-by-step removal of complex, specialized components from the CAVIS lee2025cavis model, as reported in the results in Tab. 6 of the main manuscript.
• Figure : CAVIS (15 FPS)