Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation

TL;DR

This work tackles open-world video object segmentation by introducing UVO, a taxonomy-free, densely annotated benchmark that scales beyond prior datasets. It combines a semi-automatic annotation pipeline with exhaustive framewise masks to enable evaluation of objects unseen during training, pushing beyond traditional closed-world segmentation. Through extensive baselines (top-down, bottom-up, and tracking approaches) and cross-dataset analyses, the paper demonstrates that open-world segmentation remains significantly more challenging and that annotation density and pretraining choices materially affect performance. The UVO dataset and analyses are poised to spur advances in long-term video understanding, object interactions, and flexible, category-agnostic reasoning in dynamic scenes.

Abstract

Current state-of-the-art object detection and segmentation methods work well under the closed-world assumption. This closed-world setting assumes that the list of object categories is available during training and deployment. However, many real-world applications require detecting or segmenting novel objects, i.e., object categories never seen during training. In this paper, we present, UVO (Unidentified Video Objects), a new benchmark for open-world class-agnostic object segmentation in videos. Besides shifting the problem focus to the open-world setup, UVO is significantly larger, providing approximately 8 times more videos compared with DAVIS, and 7 times more mask (instance) annotations per video compared with YouTube-VOS and YouTube-VIS. UVO is also more challenging as it includes many videos with crowded scenes and complex background motions. We demonstrated that UVO can be used for other applications, such as object tracking and super-voxel segmentation, besides open-world object segmentation. We believe that UVo is a versatile testbed for researchers to develop novel approaches for open-world class-agnostic object segmentation, and inspires new research directions towards a more comprehensive video understanding beyond classification and detection.

Figures (9)

• Figure 1: State-of-the-art object detection/segmentation methods do not work well in open-world settings. We evaluated (a) Mask R-CNN trained on COCO and (b) Google AI cloud API on Kinetics-400 videos, and found both methods fail to segment many objects that have not been seen in training. (c) Real-world applications require segmenting all objects that appear in the videos, even unseen objects. Mask R-CNN works well only on predefined categories and fails to recognize objects (e.g., barbell) or confuses non-object with objects in the taxonomy (refrigerator). Google cloud object detector offers stronger detection results but still misses all gym equipment in the background. (c) UVO is designed to detect/segment all objects regardless of the categories and beyond.
• Figure 2: Distribution of the number of objects per video. The distribution is long-tail, with a mean of 12.29 and a median of 8. In some extreme cases, we observe nearly 100 object instances in a videos. This is close to real world distribution and more suitable for open-world applications.
• Figure 3: Examples of UVO. UVO videos are exhaustively annotated with masks regardless of object categories. UVO features a wide-range of videos (e.g., third-person/egocentric, professional/amateur, crowded/sparse objects) making it a challenging benchmark. Best viewed in color.
• Figure 4: Annotation pipeline overview. We propose a semi-automated pipeline to accelerate the annotation process. We first annotate the videos sparsely (e.g., 1fps), then propagate the masks to the next frames densely. The propagated masks are then corrected by annotators.
• Figure 5: Propagating masks from sparsely annotated frames. We interpolate all the object masks between two sparsely annotated frames $t$ and $t+k$. For an un-annotated frame, we generate its annotations by forward and backward tracking the annotations from frame $t$ and $t+k$, matching and combining these two sets of annotations.