Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Local-Global Context Aware Transformer for Language-Guided Video Segmentation

Chen Liang, Wenguan Wang, Tianfei Zhou, Jiaxu Miao, Yawei Luo, Yi Yang

TL;DR

This work tackles language-guided video segmentation by introducing Locater, a memory-augmented Transformer that achieves efficient long-range temporal modeling via a finite memory consisting of global and local components. A language-guided visual encoder produces frame-wise features, which are contextually enriched by memory-based attention to yield frame-specific queries for mask decoding. The approach delivers state-of-the-art results across standard LVS benchmarks and introduces the harder A2D-S$^+$ dataset to stress grounding among semantically similar objects, ultimately achieving top performance and winning the RVOS track in YTB-VOS$^{21}$. The combination of local-global memory, frame-specific query generation, and deep supervision enables robust, scalable LVS with linear time complexity and constant memory usage, offering practical benefits for large-scale video understanding and cross-modal grounding.

Abstract

We explore the task of language-guided video segmentation (LVS). Previous algorithms mostly adopt 3D CNNs to learn video representation, struggling to capture long-term context and easily suffering from visual-linguistic misalignment. In light of this, we present Locater (local-global context aware Transformer), which augments the Transformer architecture with a finite memory so as to query the entire video with the language expression in an efficient manner. The memory is designed to involve two components -- one for persistently preserving global video content, and one for dynamically gathering local temporal context and segmentation history. Based on the memorized local-global context and the particular content of each frame, Locater holistically and flexibly comprehends the expression as an adaptive query vector for each frame. The vector is used to query the corresponding frame for mask generation. The memory also allows Locater to process videos with linear time complexity and constant size memory, while Transformer-style self-attention computation scales quadratically with sequence length. To thoroughly examine the visual grounding capability of LVS models, we contribute a new LVS dataset, A2D-S+, which is built upon A2D-S dataset but poses increased challenges in disambiguating among similar objects. Experiments on three LVS datasets and our A2D-S+ show that Locater outperforms previous state-of-the-arts. Further, we won the 1st place in the Referring Video Object Segmentation Track of the 3rd Large-scale Video Object Segmentation Challenge, where Locater served as the foundation for the winning solution. Our code and dataset are available at: https://github.com/leonnnop/Locater

Local-Global Context Aware Transformer for Language-Guided Video Segmentation

TL;DR

This work tackles language-guided video segmentation by introducing Locater, a memory-augmented Transformer that achieves efficient long-range temporal modeling via a finite memory consisting of global and local components. A language-guided visual encoder produces frame-wise features, which are contextually enriched by memory-based attention to yield frame-specific queries for mask decoding. The approach delivers state-of-the-art results across standard LVS benchmarks and introduces the harder A2D-S dataset to stress grounding among semantically similar objects, ultimately achieving top performance and winning the RVOS track in YTB-VOS. The combination of local-global memory, frame-specific query generation, and deep supervision enables robust, scalable LVS with linear time complexity and constant memory usage, offering practical benefits for large-scale video understanding and cross-modal grounding.

Abstract

We explore the task of language-guided video segmentation (LVS). Previous algorithms mostly adopt 3D CNNs to learn video representation, struggling to capture long-term context and easily suffering from visual-linguistic misalignment. In light of this, we present Locater (local-global context aware Transformer), which augments the Transformer architecture with a finite memory so as to query the entire video with the language expression in an efficient manner. The memory is designed to involve two components -- one for persistently preserving global video content, and one for dynamically gathering local temporal context and segmentation history. Based on the memorized local-global context and the particular content of each frame, Locater holistically and flexibly comprehends the expression as an adaptive query vector for each frame. The vector is used to query the corresponding frame for mask generation. The memory also allows Locater to process videos with linear time complexity and constant size memory, while Transformer-style self-attention computation scales quadratically with sequence length. To thoroughly examine the visual grounding capability of LVS models, we contribute a new LVS dataset, A2D-S+, which is built upon A2D-S dataset but poses increased challenges in disambiguating among similar objects. Experiments on three LVS datasets and our A2D-S+ show that Locater outperforms previous state-of-the-arts. Further, we won the 1st place in the Referring Video Object Segmentation Track of the 3rd Large-scale Video Object Segmentation Challenge, where Locater served as the foundation for the winning solution. Our code and dataset are available at: https://github.com/leonnnop/Locater
Paper Structure (29 sections, 11 equations, 6 figures, 10 tables)

This paper contains 29 sections, 11 equations, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language-guided Video Segmentation (LVS)
  4. Referring Image Segmentation (RIS)
  5. Referring Expression Comprehension (REC)
  6. Semi-automatic $_{\!}$Video $_{\!}$Object $_{\!}$Segmentation $_{\!}$(SVOS)
  7. Transformer in Vision-Language Tasks
  8. Transformer in Visual Referring Tasks
  9. Neural Networks with External Memory
  10. Methodology
  11. Preliminary: Transformer Architecture
  12. Local-Global Context Aware Transformer
  13. Visual-Linguistic Encoder
  14. Local-Global Memory
  15. Referring Decoder
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Our main idea. Previous LVS models are mainly built upon 3D CNNs, bounded by the local observations from short durations. They thus fail to identify the target referent in the early stage beyond the 'fall' event. In contrast, Locater is fully aware of both local and global context, enabling a holistic understanding of entire video content and correctly grounding the phrase over the whole video.
  • Figure 2: Illustration of our Locater. Building upon Transformer encoder-decoder architecture, Locater maintains a finite memory to collect and retain both global and local temporal context. By referencing the memory, Locater can reach a holistic understanding of video content and query the entire video with the expression, with linear computation complexity and constant storage consumption. See §\ref{['sec:Locater']} for details.
  • Figure 3: Attention visualization of frame-specific queries (Eq. \ref{['eq:query']}).
  • Figure 4: Representative video examples from our proposed (a) A2D-S$^+_{\text{M}}$, (b) A2D-S$^+_{\text{S}}$, and (c) A2D-S$^+_{~\!\text{T}}$ datasets (§\ref{['sec:dataset_consturction']}).
  • Figure 5: Visual comparison results (§\ref{['sec:qua_vis']}) on A2D-S testgavrilyuk2018actor (left) and R-YTVOS valseo2020urvos (right). Referents and corresponding descriptions are highlighted in the same colour. In right column, we show qualitative results of CSTM hui2021collaborative (1st row) and Locater (2nd row).
  • ...and 1 more figures