Hybrid Open-set Segmentation with Synthetic Negative Data

TL;DR

Open-set segmentation requires identifying unknown semantic regions while preserving known class labels. The authors introduce DenseHybrid, a dense anomaly detector that merges an unnormalized density $\hat{p}(\mathbf{x})$ with a discriminative dataset posterior $P(d_{in}|\mathbf{x})$, yielding a log-ratio score $s_H(\mathbf{x}) = s_D(\mathbf{x}) + s_G(\mathbf{x})$ that can override closed-set predictions. Training leverages real or synthetic negatives via a jointly trained normalizing flow, enabling end-to-end fine-tuning atop any closed-set segmentation model without sampling the intractable normalization constant $Z$. A new open-set metric, Open-IoU, quantifies the performance gap between closed-set and open-set settings. Empirical results on Fishyscapes, SMIYC, Pascal-COCO, and COCO20/80 show competitive open-set performance with negligible computational overhead, enabling practical deployment in safety-critical applications.

Abstract

Open-set segmentation can be conceived by complementing closed-set classification with anomaly detection. Many of the existing dense anomaly detectors operate through generative modelling of regular data or by discriminating with respect to negative data. These two approaches optimize different objectives and therefore exhibit different failure modes. Consequently, we propose a novel anomaly score that fuses generative and discriminative cues. Our score can be implemented by upgrading any closed-set segmentation model with dense estimates of dataset posterior and unnormalized data likelihood. The resulting dense hybrid open-set models require negative training images that can be sampled from an auxiliary negative dataset, from a jointly trained generative model, or from a mixture of both sources. We evaluate our contributions on benchmarks for dense anomaly detection and open-set segmentation. The experiments reveal strong open-set performance in spite of negligible computational overhead.

Figures (14)

• Figure 1: Open-set segmentation simultaneously classifies known scene parts and identifies unknown classes (highlighted in cyan). Our approach exploits the fact that unknown classes are semantic anomalies ruff21pieee. Hence, we construct a dense hybrid anomaly detector and use it to override closed-set decisions in anomalous pixels. Our hybrid anomaly score identifies pixels as unknown visual concepts by efficient ensembling of generative and discriminative predictions.
• Figure 2: Three anomaly detection approaches on a toy problem. Inliers, train negatives and test anomalies are shown as blue, green and red points (details in the Appendix). The background heatmaps designate the three anomaly scores with higher values in red. The discriminative anomaly score (left) is susceptible to false negative responses since the negative training dataset is finite and cannot cover all modes of test anomalies. The generative anomaly score (middle) errs along the border of the inlier manifold due to over-generalization nalisnick19iclrlucas19nips, but is unlikely to commit errors far from the inlier manifold. Our hybrid approach prevails by ensembling discriminative and generative cues.
• Figure 3: Our open-set segmentation approach complements any semantic segmentation model which recovers dense logits with our hybrid anomaly detection. Our dense anomaly score is a log-ratio of dataset posterior and data likelihood. We implement open-set segmentation by overriding the closed-set output with thresholded anomaly score.
• Figure 4: Fine-tuning procedure for the proposed DenseHybrid model. We construct mixed-content images by pasting negatives into inlier images according to (\ref{['eq:pasting']}). The negative training data can be sourced from an auxiliary real dataset (Sec. \ref{['sec:oss_train_aux']}), from a jointly trained normalizing flow (Sec. \ref{['sec:oss_train_syn']}), or from both sources according to $b$ from (\ref{['eq:neg_mix']}). Mixed-content images are fed to the open-set model that produces three dense outputs: the closed-set class posterior, unnormalized data likelihood, and dataset posterior. The model is optimized according to the compound loss (\ref{['eq:final_loss']}). In the case of synthetic negatives ($S=0$), the normalizing flow optimizes the loss (\ref{['eq:flow_total']}).
• Figure 5: We extend the standard closed-set metric (top-left) with a novel open-set metric (right). Open-IoU takes into account false positive semantics at undetected anomalies as well as false negative semantics due to false positive anomalies. The proposed open-mIoU metric quantifies dense recognition performance in the presence of anomalies.