Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What is Point Supervision Worth in Video Instance Segmentation?

Shuaiyi Huang, De-An Huang, Zhiding Yu, Shiyi Lan, Subhashree Radhakrishnan, Jose M. Alvarez, Abhinav Shrivastava, Anima Anandkumar

TL;DR

This work tackles the high annotation cost of video instance segmentation by proposing PointVIS, a point-supervised VIS framework that uses one labeled point per object per frame. It leverages class-agnostic spatio-temporal proposals generated from a COCO-pretrained image model, a spatio-temporal point-based matcher with cross-instance negatives and a maskness cue, and a self-training loop to produce dense pseudo-labels from sparse point annotations. The approach enables open-set generalization to new categories and delivers competitive performance close to fully supervised methods across three VIS benchmarks, with strong ablations validating the design choices. By substantially reducing annotation effort while maintaining accuracy, PointVIS offers a practical path toward scalable VIS in real-world, diverse environments.

Abstract

Video instance segmentation (VIS) is a challenging vision task that aims to detect, segment, and track objects in videos. Conventional VIS methods rely on densely-annotated object masks which are expensive. We reduce the human annotations to only one point for each object in a video frame during training, and obtain high-quality mask predictions close to fully supervised models. Our proposed training method consists of a class-agnostic proposal generation module to provide rich negative samples and a spatio-temporal point-based matcher to match the object queries with the provided point annotations. Comprehensive experiments on three VIS benchmarks demonstrate competitive performance of the proposed framework, nearly matching fully supervised methods.

What is Point Supervision Worth in Video Instance Segmentation?

TL;DR

This work tackles the high annotation cost of video instance segmentation by proposing PointVIS, a point-supervised VIS framework that uses one labeled point per object per frame. It leverages class-agnostic spatio-temporal proposals generated from a COCO-pretrained image model, a spatio-temporal point-based matcher with cross-instance negatives and a maskness cue, and a self-training loop to produce dense pseudo-labels from sparse point annotations. The approach enables open-set generalization to new categories and delivers competitive performance close to fully supervised methods across three VIS benchmarks, with strong ablations validating the design choices. By substantially reducing annotation effort while maintaining accuracy, PointVIS offers a practical path toward scalable VIS in real-world, diverse environments.

Abstract

Video instance segmentation (VIS) is a challenging vision task that aims to detect, segment, and track objects in videos. Conventional VIS methods rely on densely-annotated object masks which are expensive. We reduce the human annotations to only one point for each object in a video frame during training, and obtain high-quality mask predictions close to fully supervised models. Our proposed training method consists of a class-agnostic proposal generation module to provide rich negative samples and a spatio-temporal point-based matcher to match the object queries with the provided point annotations. Comprehensive experiments on three VIS benchmarks demonstrate competitive performance of the proposed framework, nearly matching fully supervised methods.
Paper Structure (20 sections, 4 equations, 4 figures, 6 tables)

This paper contains 20 sections, 4 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image instance segmentation
  4. Video Instance Segmentation
  5. Point-supervised methods.
  6. Method
  7. Problem Setup
  8. Learning VIS with Sparse Points
  9. Point-based matcher.
  10. Self-training for generalization.
  11. Experiments
  12. Experimental Setup
  13. Comparison with Fully-Supervised SOTA
  14. Ablation Study
  15. Effects of model components.
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Point-supervised video instance segmentation in this work (YoutubeVIS-2021). Top: point-level annotations in the training set (pseudo masks generated from our method overlaid); Bottom: mask predictions in the validation set.
  • Figure 2: Method Overview. Our method consists of class-agnostic spatial-temporal proposal generation, a spatio-temporal point-based matcher to match object queires with point annotations for high-quality pseudo-label generation, and self-training to mitigate the domain gap between images and videos. See text for details.
  • Figure 3: Visualization of point annotations and pseudo masks obtained by our method on Youtube-VIS 2019 yang2019video (row1), Youtube-VIS 2021 yang2019video (row2), and OVIS qi2021occluded (row3) training set.
  • Figure 4: Failure cases on OVIS qi2021occluded. We observe temporal inconsistency (e.g. tiger in the top left) or missing instances (e.g. person in white) in our pseudo masks.