A Likelihood Ratio-Based Approach to Segmenting Unknown Objects

TL;DR

An adaptive, lightweight unknown estimation module (UEM) for outlier supervision that significantly enhances the OoD segmentation performance without affecting the learned feature representation of the original network.

Abstract

Addressing the Out-of-Distribution (OoD) segmentation task is a prerequisite for perception systems operating in an open-world environment. Large foundational models are frequently used in downstream tasks, however, their potential for OoD remains mostly unexplored. We seek to leverage a large foundational model to achieve robust representation. Outlier supervision is a widely used strategy for improving OoD detection of the existing segmentation networks. However, current approaches for outlier supervision involve retraining parts of the original network, which is typically disruptive to the model's learned feature representation. Furthermore, retraining becomes infeasible in the case of large foundational models. Our goal is to retrain for outlier segmentation without compromising the strong representation space of the foundational model. To this end, we propose an adaptive, lightweight unknown estimation module (UEM) for outlier supervision that significantly enhances the OoD segmentation performance without affecting the learned feature representation of the original network. UEM learns a distribution for outliers and a generic distribution for known classes. Using the learned distributions, we propose a likelihood-ratio-based outlier scoring function that fuses the confidence of UEM with that of the pixel-wise segmentation inlier network to detect unknown objects. We also propose an objective to optimize this score directly. Our approach achieves a new state-of-the-art across multiple datasets, outperforming the previous best method by 5.74% average precision points while having a lower false-positive rate. Importantly, strong inlier performance remains unaffected.

Figures (2)

• Figure 1: Overview. Our proposed unknown estimation module (UEM) takes the input from the frozen encoder backbone and learns the outlier and inlier distributions $\tilde{p}_{\text{out}}$ and $\tilde{p}_{\text{in}}$. Then, we calculate the log-likelihood ratio by combining the outputs of UEM with the class probabilities of the inlier model $\hat{p}_{\text{in}}$.
• Figure 2: Qualitative Results on SMIYC-AT (G-G). The second column (-ID score) shows the outlier score from using the GMM without outlier supervision. The third column (OoD Score) shows the anomaly score from the fine-tuned OoD detection head. The fourth column (LR Score) shows our proposed likelihood formulation. The likelihood formulation combines information from both and predicts more accurate OoD score maps.