Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Expression Prompt Collaboration Transformer for Universal Referring Video Object Segmentation

Jiajun Chen, Jiacheng Lin, Guojin Zhong, Haolong Fu, Ke Nai, Kailun Yang, Zhiyong Li

TL;DR

The paper addresses universal referring video object segmentation by handling both audio-guided and text-guided prompts within a single framework. It introduces EPCFormer, a two-branch Transformer that processes text and audio prompts, and couples it with Expression Alignment (EA) and Expression-Visual Attention (EVA) to align modalities and enable deep tri-modal interactions with video features. EA uses contrastive learning to align audio and text semantics, while EVA provides Audio-Text Collaboration and Expression-Visual Interaction to refine referring cues and visual regions, respectively. Empirical results on multiple A-VOS and R-VOS benchmarks show state-of-the-art performance and good cross-task generalizability, underscoring the practical impact for flexible, efficient video segmentation guided by diverse expressions.

Abstract

Audio-guided Video Object Segmentation (A-VOS) and Referring Video Object Segmentation (R-VOS) are two highly related tasks that both aim to segment specific objects from video sequences according to expression prompts. However, due to the challenges of modeling representations for different modalities, existing methods struggle to strike a balance between interaction flexibility and localization precision. In this paper, we address this problem from two perspectives: the alignment of audio and text and the deep interaction among audio, text, and visual modalities. First, we propose a universal architecture, the Expression Prompt Collaboration Transformer, herein EPCFormer. Next, we propose an Expression Alignment (EA) mechanism for audio and text. The proposed EPCFormer exploits the fact that audio and text prompts referring to the same objects are semantically equivalent by using contrastive learning for both types of expressions. Then, to facilitate deep interactions among audio, text, and visual modalities, we introduce an Expression-Visual Attention (EVA) module. The knowledge of video object segmentation in terms of the expression prompts can seamlessly transfer between the two tasks by deeply exploring complementary cues between text and audio. Experiments on well-recognized benchmarks demonstrate that our EPCFormer attains state-of-the-art results on both tasks. The source code will be made publicly available at https://github.com/lab206/EPCFormer.

Expression Prompt Collaboration Transformer for Universal Referring Video Object Segmentation

TL;DR

The paper addresses universal referring video object segmentation by handling both audio-guided and text-guided prompts within a single framework. It introduces EPCFormer, a two-branch Transformer that processes text and audio prompts, and couples it with Expression Alignment (EA) and Expression-Visual Attention (EVA) to align modalities and enable deep tri-modal interactions with video features. EA uses contrastive learning to align audio and text semantics, while EVA provides Audio-Text Collaboration and Expression-Visual Interaction to refine referring cues and visual regions, respectively. Empirical results on multiple A-VOS and R-VOS benchmarks show state-of-the-art performance and good cross-task generalizability, underscoring the practical impact for flexible, efficient video segmentation guided by diverse expressions.

Abstract

Audio-guided Video Object Segmentation (A-VOS) and Referring Video Object Segmentation (R-VOS) are two highly related tasks that both aim to segment specific objects from video sequences according to expression prompts. However, due to the challenges of modeling representations for different modalities, existing methods struggle to strike a balance between interaction flexibility and localization precision. In this paper, we address this problem from two perspectives: the alignment of audio and text and the deep interaction among audio, text, and visual modalities. First, we propose a universal architecture, the Expression Prompt Collaboration Transformer, herein EPCFormer. Next, we propose an Expression Alignment (EA) mechanism for audio and text. The proposed EPCFormer exploits the fact that audio and text prompts referring to the same objects are semantically equivalent by using contrastive learning for both types of expressions. Then, to facilitate deep interactions among audio, text, and visual modalities, we introduce an Expression-Visual Attention (EVA) module. The knowledge of video object segmentation in terms of the expression prompts can seamlessly transfer between the two tasks by deeply exploring complementary cues between text and audio. Experiments on well-recognized benchmarks demonstrate that our EPCFormer attains state-of-the-art results on both tasks. The source code will be made publicly available at https://github.com/lab206/EPCFormer.
Paper Structure (22 sections, 10 equations, 8 figures, 13 tables)

This paper contains 22 sections, 10 equations, 8 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Referring Video Object Segmentation
  4. Contrastive Learning for Multi-modal Alignment
  5. Universal Visual Segmentation
  6. Method
  7. Overview
  8. Multi-modal Encoding
  9. Expression Alignment
  10. Multi-modal Interaction
  11. Segmentation and Optimization
  12. Experiments
  13. Datasets
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 7 more sections

Figures (8)

  • Figure 1: Comparison of EPCFormer and existing models. (a) A typical text-guided model for R-VOS. (b) A typical audio-guided model for A-VOS. (c) Our EPCFormer, which is capable of processing both text and audio prompts.
  • Figure 2: An illustration of the proposed EPCFormer. Its components consist of four key stages: (1) multi-modal encoding, (2) expression alignment, (3) multi-modal interaction, and (4) segmentation and optimization.
  • Figure 3: The architecture of the proposed Expression-Visual Attention (EVA) module. EVA is a two-stream structure, comprising Audio-Text Collaboration (ATC) and Expression-Visual Interaction (EVI). Given text, audio, and visual features obtained through their respective encoders, audio features, and text features are firstly integrated through linear combination, resulting in referring features. Meanwhile, the proposed ATC exploits the complementarity between text and audio features, thus facilitating the extraction of pivotal expression features. The EVI enhances visual features by incorporating referring features and, reciprocally, refines referring features through visual features.
  • Figure 4: Visualization of heat maps in the proposed EPCFomrer and Wnet pan2022wnet on A-Youtube-VOS pan2022wnetdataset. EPCFormer demonstrates the capability to discern referred objects based on audio prompts.
  • Figure 5: The quality results in EPCFomrer and Wnet pan2022wnet on A-Youtube-VOS pan2022wnet. The proposed EPCFomrer can generate more detailed masks.
  • ...and 3 more figures