Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Aleatoric Uncertainty in Object Detection via Vision Foundation Models

Peng Cui, Guande He, Dan Zhang, Zhijie Deng, Yinpeng Dong, Jun Zhu

TL;DR

This paper proposes to estimate the data uncertainty of each object instance based on the feature space of vision foundation models, which are trained on ultra-large-scale datasets and able to exhibit universal data representation and assumes a mixture-of-Gaussian structure of the object features.

Abstract

Datasets collected from the open world unavoidably suffer from various forms of randomness or noiseness, leading to the ubiquity of aleatoric (data) uncertainty. Quantifying such uncertainty is particularly pivotal for object detection, where images contain multi-scale objects with occlusion, obscureness, and even noisy annotations, in contrast to images with centric and similar-scale objects in classification. This paper suggests modeling and exploiting the uncertainty inherent in object detection data with vision foundation models and develops a data-centric reliable training paradigm. Technically, we propose to estimate the data uncertainty of each object instance based on the feature space of vision foundation models, which are trained on ultra-large-scale datasets and able to exhibit universal data representation. In particular, we assume a mixture-of-Gaussian structure of the object features and devise Mahalanobis distance-based measures to quantify the data uncertainty. Furthermore, we suggest two curial and practical usages of the estimated uncertainty: 1) for defining uncertainty-aware sample filter to abandon noisy and redundant instances to avoid over-fitting, and 2) for defining sample adaptive regularizer to balance easy/hard samples for adaptive training. The estimated aleatoric uncertainty serves as an extra level of annotations of the dataset, so it can be utilized in a plug-and-play manner with any model. Extensive empirical studies verify the effectiveness of the proposed aleatoric uncertainty measure on various advanced detection models and challenging benchmarks.

Exploring Aleatoric Uncertainty in Object Detection via Vision Foundation Models

TL;DR

This paper proposes to estimate the data uncertainty of each object instance based on the feature space of vision foundation models, which are trained on ultra-large-scale datasets and able to exhibit universal data representation and assumes a mixture-of-Gaussian structure of the object features.

Abstract

Datasets collected from the open world unavoidably suffer from various forms of randomness or noiseness, leading to the ubiquity of aleatoric (data) uncertainty. Quantifying such uncertainty is particularly pivotal for object detection, where images contain multi-scale objects with occlusion, obscureness, and even noisy annotations, in contrast to images with centric and similar-scale objects in classification. This paper suggests modeling and exploiting the uncertainty inherent in object detection data with vision foundation models and develops a data-centric reliable training paradigm. Technically, we propose to estimate the data uncertainty of each object instance based on the feature space of vision foundation models, which are trained on ultra-large-scale datasets and able to exhibit universal data representation. In particular, we assume a mixture-of-Gaussian structure of the object features and devise Mahalanobis distance-based measures to quantify the data uncertainty. Furthermore, we suggest two curial and practical usages of the estimated uncertainty: 1) for defining uncertainty-aware sample filter to abandon noisy and redundant instances to avoid over-fitting, and 2) for defining sample adaptive regularizer to balance easy/hard samples for adaptive training. The estimated aleatoric uncertainty serves as an extra level of annotations of the dataset, so it can be utilized in a plug-and-play manner with any model. Extensive empirical studies verify the effectiveness of the proposed aleatoric uncertainty measure on various advanced detection models and challenging benchmarks.

Paper Structure

This paper contains 22 sections, 9 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Feature space density modeling.
  4. Aleatoric uncertainty.
  5. Aleatoric Uncertainty Quantification in Object Detection
  6. Multivariate Gaussian distribution.
  7. Mahalanobis distance-based uncertainty score.
  8. A Recipe for Aleatoric Uncertainty in Object Detection
  9. Aleatoric uncertainty-aware data filtering
  10. Filtering out noisy objects.
  11. Filtering out redundant objects.
  12. Aleatoric uncertainty-aware Regularization
  13. Experiments
  14. Performance on uncertainty-aware regularizer
  15. Performance on uncertainty-aware data filter
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Visualization of scoring objects with corresponding uncertainty scores in training images of MS-COCO lin2014microsoft. The orange bounding box is the annotated ground truth. "Hard" objects suffer from occlusion or obscureness within an image, and "noisy" ones have misleading bounding boxes.
  • Figure 2: The histogram of ${d}(z_j|c_j)$ for MS-COCO.
  • Figure A1: Distributions of data uncertainty scores for different classes.
  • Figure A2: Visualization of scoring objects with corresponding uncertainty scores in training images of MS-COCO lin2014microsoft for class "dog" and "bird".
  • Figure A3: Visualization of scoring objects with corresponding uncertainty scores from training images of Pascal VOC Everingham15 for class "person" and "car".
  • ...and 1 more figures