Segmenting Known Objects and Unseen Unknowns without Prior Knowledge

TL;DR

The paper defines holistic segmentation as segmenting unseen unknown objects into instances while performing panoptic segmentation on known classes, without relying on prior knowledge about unknowns. It proposes U3HS, a framework that uses uncertainty estimation to locate unknown regions and learning of instance-aware embeddings trained only on known objects, followed by DBSCAN clustering to form unknown instances. U3HS leverages a dedicated prototype/embedding architecture and Dirichlet-based semantic uncertainty to separate known and unknown regions, achieving strong PQ performance on unseen categories across Cityscapes-derived Lost&Found and MS COCO, without training on unknown classes. The approach offers a practical, unconstrained path toward safer perception in safety-critical systems by identifying and tracking entirely unknown objects without external priors.

Abstract

Panoptic segmentation methods assign a known class to each pixel given in input. Even for state-of-the-art approaches, this inevitably enforces decisions that systematically lead to wrong predictions for objects outside the training categories. However, robustness against out-of-distribution samples and corner cases is crucial in safety-critical settings to avoid dangerous consequences. Since real-world datasets cannot contain enough data points to adequately sample the long tail of the underlying distribution, models must be able to deal with unseen and unknown scenarios as well. Previous methods targeted this by re-identifying already-seen unlabeled objects. In this work, we propose the necessary step to extend segmentation with a new setting which we term holistic segmentation. Holistic segmentation aims to identify and separate objects of unseen, unknown categories into instances without any prior knowledge about them while performing panoptic segmentation of known classes. We tackle this new problem with U3HS, which finds unknowns as highly uncertain regions and clusters their corresponding instance-aware embeddings into individual objects. By doing so, for the first time in panoptic segmentation with unknown objects, our U3HS is trained without unknown categories, reducing assumptions and leaving the settings as unconstrained as in real-life scenarios. Extensive experiments on public data from MS COCO, Cityscapes, and Lost&Found demonstrate the effectiveness of U3HS for this new, challenging, and assumptions-free setting called holistic segmentation. Project page: https://holisticseg.github.io.

Figures (13)

• Figure 1: State-of-the-art panoptic segmentation methods cheng2020panopticdeeplab cannot deal with unseen classes from pinggera2016lost (top right). Instead, our U3HS addresses the proposed holistic segmentation setting. U3HS finds unseen unknowns (bottom left) and separates them into instances (bottom right) without prior knowledge about unknowns.
• Figure 2: Comparison between closed-set (top right) kirillov2019panoptic and open-set hwang2021exemplar panoptic segmentation, zero-shot learning zheng2021zero_instance, and the proposed holistic segmentation setting. While zero-shot and open-set panoptic methods commonly leverage knowledge about unknown objects, holistic segmentation does not use any priors.
• Figure 3: The proposed U3HS framework. Uncertainty is estimated in the semantic branch, and with the instance-aware embeddings, it determines unknown instances. Known instances are found via center regression and formed by grouping embeddings with their prototypes.
• Figure 4: Example predictions of U3HS on OOD data from the Lost&Found pinggera2016lost test set. The model was trained on Cityscapes cordts2016cityscapes and transferred to Lost&Found without fine-tuning. Embeddings are projected to RGB via t-SNE van2008tsne. White arrows mark labeled unknowns.
• Figure 5: Example predictions of U3HS on OOD data from the COCO lin2014coco validation set. The model had never seen images containing bear or frisbee (part of the held-out classes), nor had any information about them. Colors represent the predicted instances.