Uncertainty estimation in Deep Learning for Panoptic segmentation

TL;DR

This paper demonstrates how ensemble-based uncertainty estimation approaches such as Monte Carlo Dropout can be used in the panoptic segmentation domain with no changes to an existing network, providing both improved performance and more importantly a better measure of uncertainty for predictions made by the network.

Abstract

As deep learning-based computer vision algorithms continue to advance the state of the art, their robustness to real-world data continues to be an issue, making it difficult to bring an algorithm from the lab to the real world. Ensemble-based uncertainty estimation approaches such as Monte Carlo Dropout have been successfully used in many applications in an attempt to address this robustness issue. Unfortunately, it is not always clear if such ensemble-based approaches can be applied to a new problem domain. This is the case with panoptic segmentation, where the structure of the problem and architectures designed to solve it means that unlike image classification or even semantic segmentation, the typical solution of using a mean across samples cannot be directly applied. In this paper, we demonstrate how ensemble-based uncertainty estimation approaches such as Monte Carlo Dropout can be used in the panoptic segmentation domain with no changes to an existing network, providing both improved performance and more importantly a better measure of uncertainty for predictions made by the network. Results are demonstrated quantitatively and qualitatively on the COCO, KITTI-STEP and VIPER datasets.

Figures (8)

• Figure 1: Two sequential frames (cropped for clarity) from the KITTI-STEP weber_step_2021 dataset. Note the change from identification of the oncoming vehicle from car to truck, and how our approach better captures the uncertainty in the prediction.
• Figure 2: Overview of our approach. See text for details. Note that for conciseness and clarity, some implementation details have been simplified.
• Figure 3: Binning and plotting predictive entropy (ours) or softmax entropy (baseline) for our approach with Monte Carlo dropout and the baseline respectively. Top: clean data. Bottom: Gaussian and Shot noise at high severity. Note that the y-axis follows log scale.
• Figure 4: Qualitative examples from the COCO (top 2 rows), KITTI-STEP (middle 2 rows) and VIPER (bottom 2 rows) validation sets. Left to right: input image, ground truth, baseline segmentation, our approach with 15 Monte Carlo samples.
• Figure 5: Side-by-side comparisons of predictive entropy uncertainty heatmaps for the baseline (left) and our approach with 15 MC samples (right) for the top-most example shown in \ref{['fig:qualitative_examples']}. Lighter shades indicate greater uncertainty.