Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Visual Query Segmentation in the Wild

Bing Fan, Minghao Li, Hanzhi Zhang, Shaohua Dong, Naga Prudhvi Mareedu, Weishi Shi, Yunhe Feng, Yan Huang, Heng Fan

TL;DR

A simple yet effective method is presented, named VQ-SAM, which extends SAM 2 by leveraging target-specific and background distractor cues from the video to progressively evolve the memory through a novel multi-stage framework with an adaptive memory generation (AMG) module for VQS, significantly improving the performance.

Abstract

In this paper, we introduce visual query segmentation (VQS), a new paradigm of visual query localization (VQL) that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query. Compared to existing VQL locating only the last appearance of a target using bounding boxes, VQS enables more comprehensive (i.e., all object occurrences) and precise (i.e., pixel-level masks) localization, making it more practical for real-world scenarios. To foster research on this task, we present VQS-4K, a large-scale benchmark dedicated to VQS. Specifically, VQS-4K contains 4,111 videos with more than 1.3 million frames and covers a diverse set of 222 object categories. Each video is paired with a visual query defined by a frame outside the search video and its target mask, and annotated with spatial-temporal masklets corresponding to the queried target. To ensure high quality, all videos in VQS-4K are manually labeled with meticulous inspection and iterative refinement. To the best of our knowledge, VQS-4K is the first benchmark specifically designed for VQS. Furthermore, to stimulate future research, we present a simple yet effective method, named VQ-SAM, which extends SAM 2 by leveraging target-specific and background distractor cues from the video to progressively evolve the memory through a novel multi-stage framework with an adaptive memory generation (AMG) module for VQS, significantly improving the performance. In our extensive experiments on VQS-4K, VQ-SAM achieves promising results and surpasses all existing approaches, demonstrating its effectiveness. With the proposed VQS-4K and VQ-SAM, we expect to go beyond the current VQL paradigm and inspire more future research and practical applications on VQS. Our benchmark, code, and results will be made publicly available.

Towards Visual Query Segmentation in the Wild

TL;DR

A simple yet effective method is presented, named VQ-SAM, which extends SAM 2 by leveraging target-specific and background distractor cues from the video to progressively evolve the memory through a novel multi-stage framework with an adaptive memory generation (AMG) module for VQS, significantly improving the performance.

Abstract

In this paper, we introduce visual query segmentation (VQS), a new paradigm of visual query localization (VQL) that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query. Compared to existing VQL locating only the last appearance of a target using bounding boxes, VQS enables more comprehensive (i.e., all object occurrences) and precise (i.e., pixel-level masks) localization, making it more practical for real-world scenarios. To foster research on this task, we present VQS-4K, a large-scale benchmark dedicated to VQS. Specifically, VQS-4K contains 4,111 videos with more than 1.3 million frames and covers a diverse set of 222 object categories. Each video is paired with a visual query defined by a frame outside the search video and its target mask, and annotated with spatial-temporal masklets corresponding to the queried target. To ensure high quality, all videos in VQS-4K are manually labeled with meticulous inspection and iterative refinement. To the best of our knowledge, VQS-4K is the first benchmark specifically designed for VQS. Furthermore, to stimulate future research, we present a simple yet effective method, named VQ-SAM, which extends SAM 2 by leveraging target-specific and background distractor cues from the video to progressively evolve the memory through a novel multi-stage framework with an adaptive memory generation (AMG) module for VQS, significantly improving the performance. In our extensive experiments on VQS-4K, VQ-SAM achieves promising results and surpasses all existing approaches, demonstrating its effectiveness. With the proposed VQS-4K and VQ-SAM, we expect to go beyond the current VQL paradigm and inspire more future research and practical applications on VQS. Our benchmark, code, and results will be made publicly available.
Paper Structure (16 sections, 13 equations, 11 figures, 12 tables)

This paper contains 16 sections, 13 equations, 11 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed VQS-4K
  4. Design Principle
  5. Data Acquisition
  6. Data Annotation
  7. Dataset Split and Evaluation Metric
  8. Methodology: A New Baseline for VQS
  9. The VQ-SAM Framework
  10. Adaptive Memory Generation (AMG)
  11. Optimization and Inference
  12. Experiments
  13. State-of-the-art Comparison
  14. Ablation Studies
  15. Comparison with Existing VQL Methods on VQ2D
  16. ...and 1 more sections

Figures (11)

  • Figure 1: Comparison of existing VQL paradigm that localizes the last target occurrence using box (a) and our VQS segmenting all pixel-level occurrences of an object (b).
  • Figure 2: Representative statistics on VQS-4K, including distributions of video length, response length, object occurrences, and object mask areas. Please note, values greater than 80 in video length are grouped into the final bin for visualization purposes.
  • Figure 3: Overview of VQ-SAM that mines target and background distractor information in the video via a progressive framework for VQS. Please note that, the last stage $K$ outputs $S_{K}$ for the final prediction, and TFG, DFG, and AMG modules are removed.
  • Figure 4: Architecture of our AMG.
  • Figure 5: Category organization of VQS-4K. The inner part of chart displays 19 coarser classes, while the outer part shows 222 fine categories. Best viewed by zooming in.
  • ...and 6 more figures