Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost-Sensitive Uncertainty-Based Failure Recognition for Object Detection

Moussa Kassem Sbeyti, Michelle Karg, Christian Wirth, Nadja Klein, Sahin Albayrak

TL;DR

This work proposes a cost-sensitive framework for object detection tailored to user-defined budgets on the two types of errors, missing and false detections, and automate and optimize the thresholding process to maximize the failure recognition rate w.r.t. the specified budget.

Abstract

Object detectors in real-world applications often fail to detect objects due to varying factors such as weather conditions and noisy input. Therefore, a process that mitigates false detections is crucial for both safety and accuracy. While uncertainty-based thresholding shows promise, previous works demonstrate an imperfect correlation between uncertainty and detection errors. This hinders ideal thresholding, prompting us to further investigate the correlation and associated cost with different types of uncertainty. We therefore propose a cost-sensitive framework for object detection tailored to user-defined budgets on the two types of errors, missing and false detections. We derive minimum thresholding requirements to prevent performance degradation and define metrics to assess the applicability of uncertainty for failure recognition. Furthermore, we automate and optimize the thresholding process to maximize the failure recognition rate w.r.t. the specified budget. Evaluation on three autonomous driving datasets demonstrates that our approach significantly enhances safety, particularly in challenging scenarios. Leveraging localization aleatoric uncertainty and softmax-based entropy only, our method boosts the failure recognition rate by 36-60\% compared to conventional approaches. Code is available at https://mos-ks.github.io/publications.

Cost-Sensitive Uncertainty-Based Failure Recognition for Object Detection

TL;DR

This work proposes a cost-sensitive framework for object detection tailored to user-defined budgets on the two types of errors, missing and false detections, and automate and optimize the thresholding process to maximize the failure recognition rate w.r.t. the specified budget.

Abstract

Object detectors in real-world applications often fail to detect objects due to varying factors such as weather conditions and noisy input. Therefore, a process that mitigates false detections is crucial for both safety and accuracy. While uncertainty-based thresholding shows promise, previous works demonstrate an imperfect correlation between uncertainty and detection errors. This hinders ideal thresholding, prompting us to further investigate the correlation and associated cost with different types of uncertainty. We therefore propose a cost-sensitive framework for object detection tailored to user-defined budgets on the two types of errors, missing and false detections. We derive minimum thresholding requirements to prevent performance degradation and define metrics to assess the applicability of uncertainty for failure recognition. Furthermore, we automate and optimize the thresholding process to maximize the failure recognition rate w.r.t. the specified budget. Evaluation on three autonomous driving datasets demonstrates that our approach significantly enhances safety, particularly in challenging scenarios. Leveraging localization aleatoric uncertainty and softmax-based entropy only, our method boosts the failure recognition rate by 36-60\% compared to conventional approaches. Code is available at https://mos-ks.github.io/publications.
Paper Structure (13 sections, 3 theorems, 7 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 13 sections, 3 theorems, 7 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cost-Sensitive Detection
  4. Budget and Cost-Sensitivity
  5. Thresholding Requirements
  6. Thresholding Automation and Evaluation
  7. Thresholding Optimization
  8. Experiments
  9. Uncertainty Estimation Methods
  10. Uncertainty-based Thresholding
  11. Budget Behavior Analysis
  12. Optimized Combined Thresholding
  13. Conclusion

Key Result

Theorem 1

The optimal uncertainty threshold $\delta_{\hbox{\scriptsize{opt}}}(b,\tau)\in \mathbb{R}^+$ either maximizes the TPR while the FPR is bound by $b=i$, or minimizes the FPR while the TPR is bound by $b=m$. It is used to calculate a distinct operating point $(\text{FPR}(\delta(b,\tau)), \text{TPR}(\de

Figures (6)

  • Figure 1: Illustration of the two cost-sensitive use-cases on the ROC curve: Fixing the reduction in $\mathrm{CD}$s and therefore increase in $\mathrm{MD}$s via FPR (red, $1-b=1-i=0.05$) or the reduction in $\mathrm{FD}$s via TPR (green, $b=m=0.95$).
  • Figure 2: Failure case recognition process via cost-sensitive automated and optimized uncertainty-based thresholding. Circle size symbolizes the typical occurrence rate in well-trained detectors. Dashed circles indicate original detections, solid circles represent remaining detections, and donut-shaped circles signify removed detections (consider grey circles as the legend).
  • Figure 3: KITTI (top) and BDD (bottom): Comparison between the separation ability of $\sigma$ types based on JSD and AUC (right). On the left is $\mu_\sigma\pm\sigma_\sigma$ of $\mathrm{CD}$s and $\mathrm{FD}$s.
  • Figure 4: KITTI (left) and BDD (right): Recognition rates of $\mathrm{FD}$s for a fixed budget of 95% $\mathrm{CD}$s. The circles and thicker lines indicate requirement fulfillment in \ref{['eq:f1bb', 'eq:f1bc']}.
  • Figure 5: KITTI (left), BDD (mid), CODA (right): Budget effect on the thresholding performance of the $\sigma$ types for different $b$ (%) in both use-cases. Maximum FD@CD($b$) is accentuated for comparison.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Lemma 1
  • Theorem 2
  • Remark 1