LiVOS: Light Video Object Segmentation with Gated Linear Matching

TL;DR

LiVOS is proposed, a lightweight memory network that employs linear matching via linear attention, reformulating memory matching into a recurrent process that reduces the quadratic attention matrix to a constant-size, spatiotemporal-agnostic 2D state to enhance selectivity.

Abstract

Semi-supervised video object segmentation (VOS) has been largely driven by space-time memory (STM) networks, which store past frame features in a spatiotemporal memory to segment the current frame via softmax attention. However, STM networks face memory limitations due to the quadratic complexity of softmax matching, restricting their applicability as video length and resolution increase. To address this, we propose LiVOS, a lightweight memory network that employs linear matching via linear attention, reformulating memory matching into a recurrent process that reduces the quadratic attention matrix to a constant-size, spatiotemporal-agnostic 2D state. To enhance selectivity, we introduce gated linear matching, where a data-dependent gate matrix is multiplied with the state matrix to control what information to retain or discard. Experiments on diverse benchmarks demonstrated the effectiveness of our method. It achieved 64.8 J&F on MOSE and 85.1 J&F on DAVIS, surpassing all non-STM methods and narrowing the gap with STM-based approaches. For longer and higher-resolution videos, it matched STM-based methods with 53% less GPU memory and supports 4096p inference on a 32G consumer-grade GPU--a previously cost-prohibitive capability--opening the door for long and high-resolution video foundation models.

Figures (4)

• Figure 1: Top: Conceptual comparison of softmax vs. linear matching in video object segmentation. Bottom: Softmax matching suffers from memory constraints due to its quadratic attention complexity, while linear matching achieves linear growth with a constant-size state. Curves are based on results in Tab. \ref{['tab:quantitative_highres_videos']}.
• Figure 2: CPU latency comparison between softmax matching and linear matching. Softmax attention scales linearly over time (i.e., the number of memory frames $T$) and quadratically with input resolution $HW$. Latency is measured on an Intel Core-i7 (2.80GHz) CPU with PyTorch 2.0, batch size 1, and fp32.
• Figure 3: Masks of thin structures at different resolutions. Thin structures may lose fine details at 480p, the standard resolution for VOS. However, existing memory networks relying on softmax matching are not efficient for processing high resolution videos.
• Figure 4: LiVOS Overview. Given a query frame, we first extract its key using an image encoder and retrieve its value via gated linear matching. This value is then enhanced by two external memories before being processed by a lightweight mask decoder for segmentation. Notably, during memory matching, our method requires only a constant-size state and gate matrix. The gate matrix is generated by a gate projector that converts the pixel-level features from the last memory frame into a sparse vector, which is then transformed to a gate matrix for element-wise multiplication with the state.