Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

Ci-Siang Lin, I-Jieh Liu, Min-Hung Chen, Chien-Yi Wang, Sifei Liu, Yu-Chiang Frank Wang

TL;DR

This work aims to efficiently adapt foundation segmentation models for addressing RVOS from weak supervision with the proposed Grounded Prompting (GroPrompt) framework and proposes Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences with only box supervisions.

Abstract

Referring Video Object Segmentation (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. Most existing methods require end-to-end training with dense mask annotations, which could be computation-consuming and less scalable. In this work, we aim to efficiently adapt foundation segmentation models for addressing RVOS from weak supervision with the proposed Grounded Prompting (GroPrompt) framework. More specifically, we propose Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences with only box supervisions, including Text-Contrastive Prompt Learning (TextCon) and Modality-Contrastive Prompt Learning (ModalCon) at frame level and video level, respectively. With the proposed TAP-CL, our GroPrompt framework can generate temporal-consistent yet text-aware position prompts describing locations and movements for the referred object from the video. The experimental results in the standard RVOS benchmarks (Ref-YouTube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences) demonstrate the competitive performance of our proposed GroPrompt framework given only bounding box weak supervisions.

GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

TL;DR

This work aims to efficiently adapt foundation segmentation models for addressing RVOS from weak supervision with the proposed Grounded Prompting (GroPrompt) framework and proposes Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences with only box supervisions.

Abstract

Referring Video Object Segmentation (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. Most existing methods require end-to-end training with dense mask annotations, which could be computation-consuming and less scalable. In this work, we aim to efficiently adapt foundation segmentation models for addressing RVOS from weak supervision with the proposed Grounded Prompting (GroPrompt) framework. More specifically, we propose Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences with only box supervisions, including Text-Contrastive Prompt Learning (TextCon) and Modality-Contrastive Prompt Learning (ModalCon) at frame level and video level, respectively. With the proposed TAP-CL, our GroPrompt framework can generate temporal-consistent yet text-aware position prompts describing locations and movements for the referred object from the video. The experimental results in the standard RVOS benchmarks (Ref-YouTube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences) demonstrate the competitive performance of our proposed GroPrompt framework given only bounding box weak supervisions.
Paper Structure (22 sections, 4 equations, 3 figures, 6 tables)

This paper contains 22 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Referring Video Object Segmentation
  4. Foundation Segmentation Models
  5. Proposed Method
  6. Overview
  7. Problem Definition.
  8. Framework Overview.
  9. Efficient Grounded Prompting and Adaptation
  10. Weakly-Supervised Position Prompts
  11. Text-Aware Prompt Contrastive Learning
  12. Text-Contrastive Prompt Learning.
  13. Modality-Contrastive Prompt Learning.
  14. Experiments
  15. Datasets and Evaluation Metrics
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Overview of our proposed GroPrompt framework. In (a), our proposal generation takes each frame $I_t$ and the referring sentence $S^i$ to derive object queries $Q_t^i$ and produce the prompt embedding $p_t^i$ for segmentation, with another sentence $S^j$ as input for performing Text-Contrastive Prompt Learning. In (b), to handle sentence descriptions containing long-term motions or actions in referring video object segmentation, we uniquely present Modality-Contrastive Prompt Learning to align the text with the referred object at the video level.
  • Figure 2: Qualitative comparisons of the state-of-the-art methods on Refer-DAVIS$_{17}$, where "GT-bbox + SAM" represents the result by taking ground-truth bounding boxes to prompt SAM.
  • Figure 3: Qualitative comparisons of the state-of-the-art methods on Refer-Youtube-VOS.