Criteria for Uncertainty-based Corner Cases Detection in Instance Segmentation

TL;DR

This work tackles uncertainty-driven corner-case detection for object instance segmentation in automated driving. It develops a GT-free framework based on MC-Dropout sampling to approximate predictive distributions, from which per-detection uncertainty features are extracted across class scores, bounding boxes, and instance masks. A comprehensive corner-case decision function classifies detections into TP, L-CC, C-CC, LC-CC, or FP, enabled by single and combined criteria and validated on COCO and NuImages; an iterative data-reduction cycle demonstrates annotation-cost savings while improving performance. The approach highlights specific uncertainty signals (e.g., Box/Mask IoU distributions and divergences) that correlate with localization and classification errors, offering a practical path to targeted data collection and retraining for high-stakes perception systems.

Abstract

The operating environment of a highly automated vehicle is subject to change, e.g., weather, illumination, or the scenario containing different objects and other participants in which the highly automated vehicle has to navigate its passengers safely. These situations must be considered when developing and validating highly automated driving functions. This already poses a problem for training and evaluating deep learning models because without the costly labeling of thousands of recordings, not knowing whether the data contains relevant, interesting data for further model training, it is a guess under which conditions and situations the model performs poorly. For this purpose, we present corner case criteria based on the predictive uncertainty. With our corner case criteria, we are able to detect uncertainty-based corner cases of an object instance segmentation model without relying on ground truth (GT) data. We evaluated each corner case criterion using the COCO and the NuImages dataset to analyze the potential of our approach. We also provide a corner case decision function that allows us to distinguish each object into True Positive (TP), localization and/or classification corner case, or False Positive (FP). We also present our first results of an iterative training cycle that outperforms the baseline and where the data added to the training dataset is selected based on the corner case decision function.

Figures (9)

• Figure 1: Overall approach to derive and apply uncertainty-based corner case criteria.
• Figure 2: Our approach assumes that the object instance segmentation model plus a possible post-process (gray) provides several predictions, consisting of class score $c$, bounding box $b$, and instance mask $m$, per object detection $D_{i}$. The prediction variance is used to determine the uncertainty in the ML model. The uncertainty analysis is utilized in the class score, bounding box, and instance mask criteria (single knowledge criteria -- blue) or combined in the bounding box & mask criteria (combined knowledge criteria -- magenta). The values from the criteria are interpreted as features to assign the object detections $D_{i}$ with a decision function to one of the defined corner case categories. The defined categories are True Positive (TP -- green), False Positive (FP -- red), and the corner cases (yellow) for location (L-CC), class (C-CC), and both (LC-CC).
• Figure 3: Bounding box uncertainty. The standard deviation $\sigma_{b}$ of each bounding box edge is represented by red lines, and red dots represent the spread of the bounding box center point. The blue box with the truck in the middle represents the mean bounding box.
• Figure 4: Instance mask uncertainty example.
• Figure 5: Distribution of the IoU between the mean bounding box $\overline{D}_{b}$ (blue) or mean mask $\overline{D}_{m}$ (orange) and all other bounding box $b_{j}$ or instance mask $m_{j}$.