A Dataset for Semantic Segmentation in the Presence of Unknowns

TL;DR

ISSU addresses the gap in evaluating semantic segmentation under unknowns by providing a real-world, open-set annotated dataset with both closed-set and anomaly labels. It comprises ISSU-Train, ISSU-Test-Static, and ISSU-Test-Temporal collected from Indian roads, enabling in-domain and cross-domain, as well as cross-sensor and temporal evaluations. The paper benchmarks multiple pixel- and mask-level baselines, conducts extensive ablations on anomaly size and lighting, and demonstrates significant generalization gaps under domain shifts and sensor changes. The dataset's static and temporal splits, along with ROI-focused anomaly labeling, facilitate robust evaluation and spur development of domain-general anomaly segmentation methods with practical impact for safety-critical driving.

Abstract

Before deployment in the real-world deep neural networks require thorough evaluation of how they handle both knowns, inputs represented in the training data, and unknowns (anomalies). This is especially important for scene understanding tasks with safety critical applications, such as in autonomous driving. Existing datasets allow evaluation of only knowns or unknowns - but not both, which is required to establish "in the wild" suitability of deep neural network models. To bridge this gap, we propose a novel anomaly segmentation dataset, ISSU, that features a diverse set of anomaly inputs from cluttered real-world environments. The dataset is twice larger than existing anomaly segmentation datasets, and provides a training, validation and test set for controlled in-domain evaluation. The test set consists of a static and temporal part, with the latter comprised of videos. The dataset provides annotations for both closed-set (knowns) and anomalies, enabling closed-set and open-set evaluation. The dataset covers diverse conditions, such as domain and cross-sensor shift, illumination variation and allows ablation of anomaly detection methods with respect to these variations. Evaluation results of current state-of-the-art methods confirm the need for improvements especially in domain-generalization, small and large object segmentation.

Figures (15)

• Figure 1: Standard benchmarks cannot separate the effects of domain shift, lighting conditions, and anomaly size during evaluation. The proposed dataset allows controlled evaluation of these effects and supports evaluation of both closed-set and anomaly segmentation.
• Figure 2: Examples of anomalies (shown in white) in the ISSU dataset. The anomalous examples are ordered from small (left) to very large (right). Top: examples of anomalies of different size and shape at approximately the same distance from the ego-vehicle in ISSU-Test-Static. Bottom: temporal view of an anomaly observed at different time-steps in ISSU-Test-Temporal.
• Figure 3: Distributions of anomalies with respect to their size and spatial location within images. The anomalies are quantized to four different size intervals that are used in the ablation. Anomalies less than $7\times7$ (black dashed line) are ignored during all evaluations. The spatial distributions are visualized as a probability heatmap for each image location. Green line outlines road pixels that appeared in more than $50\%$ of dataset images. For temporal dataset the spatial distribution is also visualized for different view-points.
• Figure 4: Cross-domain vs. In-domain performance in road anomaly evaluation protocol. Top row -- Static, bottom row -- Temporal. Mask$^*$ are mask-based methods trained with OOD data. The $y = x$ reference line shows relative gain or drop. The T ( F) subscript for oIoU metric refers to operating point (anomaly score threshold) for which the methods achieves $95\%$ TPR ($5\%$ FPR).
• Figure 5: Ablation of different anomaly sizes. The plot shows results in ISSU-Test-Static (left) and ISSU-Test-Temporal(right) for road anomaly evaluation protocol under in-domain setup.