Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach

Chen-Chen Zong, Sheng-Jun Huang

TL;DR

The paper tackles active open-set annotation by clarifying that relying solely on epistemic or aleatoric uncertainty is suboptimal in open-world settings. It introduces Energy-based Active Open-set Annotation (EAOA), pairing a $(C+1)$-class detector with a $C$-class target classifier and defining energy-based epistemic and aleatoric uncertainties, augmented by a margin-based energy loss and a target-driven adaptive sampling strategy. Empirical results on CIFAR-10/100 and Tiny-ImageNet show state-of-the-art performance in test accuracy and query precision with low overhead, validating the effectiveness of jointly leveraging EU and AU. The work offers practical implications for labeling efficiency in real-world open-set scenarios and provides code to foster reproducibility and further development.

Abstract

Active learning (AL), which iteratively queries the most informative examples from a large pool of unlabeled candidates for model training, faces significant challenges in the presence of open-set classes. Existing methods either prioritize query examples likely to belong to known classes, indicating low epistemic uncertainty (EU), or focus on querying those with highly uncertain predictions, reflecting high aleatoric uncertainty (AU). However, they both yield suboptimal performance, as low EU corresponds to limited useful information, and closed-set AU metrics for unknown class examples are less meaningful. In this paper, we propose an Energy-based Active Open-set Annotation (EAOA) framework, which effectively integrates EU and AU to achieve superior performance. EAOA features a $(C+1)$-class detector and a target classifier, incorporating an energy-based EU measure and a margin-based energy loss designed for the detector, alongside an energy-based AU measure for the target classifier. Another crucial component is the target-driven adaptive sampling strategy. It first forms a smaller candidate set with low EU scores to ensure closed-set properties, making AU metrics meaningful. Subsequently, examples with high AU scores are queried to form the final query set, with the candidate set size adjusted adaptively. Extensive experiments show that EAOA achieves state-of-the-art performance while maintaining high query precision and low training overhead. The code is available at https://github.com/chenchenzong/EAOA.

Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach

TL;DR

The paper tackles active open-set annotation by clarifying that relying solely on epistemic or aleatoric uncertainty is suboptimal in open-world settings. It introduces Energy-based Active Open-set Annotation (EAOA), pairing a -class detector with a -class target classifier and defining energy-based epistemic and aleatoric uncertainties, augmented by a margin-based energy loss and a target-driven adaptive sampling strategy. Empirical results on CIFAR-10/100 and Tiny-ImageNet show state-of-the-art performance in test accuracy and query precision with low overhead, validating the effectiveness of jointly leveraging EU and AU. The work offers practical implications for labeling efficiency in real-world open-set scenarios and provides code to foster reproducibility and further development.

Abstract

Active learning (AL), which iteratively queries the most informative examples from a large pool of unlabeled candidates for model training, faces significant challenges in the presence of open-set classes. Existing methods either prioritize query examples likely to belong to known classes, indicating low epistemic uncertainty (EU), or focus on querying those with highly uncertain predictions, reflecting high aleatoric uncertainty (AU). However, they both yield suboptimal performance, as low EU corresponds to limited useful information, and closed-set AU metrics for unknown class examples are less meaningful. In this paper, we propose an Energy-based Active Open-set Annotation (EAOA) framework, which effectively integrates EU and AU to achieve superior performance. EAOA features a -class detector and a target classifier, incorporating an energy-based EU measure and a margin-based energy loss designed for the detector, alongside an energy-based AU measure for the target classifier. Another crucial component is the target-driven adaptive sampling strategy. It first forms a smaller candidate set with low EU scores to ensure closed-set properties, making AU metrics meaningful. Subsequently, examples with high AU scores are queried to form the final query set, with the candidate set size adjusted adaptively. Extensive experiments show that EAOA achieves state-of-the-art performance while maintaining high query precision and low training overhead. The code is available at https://github.com/chenchenzong/EAOA.

Paper Structure

This paper contains 16 sections, 15 equations, 10 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Energy-based Epistemic and Aleatoric Uncertainty for Active Sampling
  6. Detector and Target Classifier Training
  7. Experiments
  8. Implementation Details
  9. Performance Comparison
  10. Ablation Studies
  11. Conclusion
  12. Uncertainty Quantification
  13. Label-Wise Free Energy
  14. The Pseudocode of EAOA
  15. Comparing Methods
  16. ...and 1 more sections

Figures (10)

  • Figure 1: Dataset: CIFAR-10; mismatch ratio: 40%. Our motivation: in open-set scenarios, querying examples with low epistemic uncertainty yields high query precision, but the overall information content is low, resulting in poor model performance. Focusing on examples with high aleatoric uncertainty also leads to poor performance, as the model's assessment of this uncertainty becomes meaningless for open-set examples. Nevertheless, an effective combination of the two can lead to a superior outcome.
  • Figure 2: The framework of EAOA. It consists of three general steps: model training, example selection, and Oracle labeling. In the model training phase, a detector is trained to assess epistemic uncertainty (EU) from both learning-based and data-driven perspectives, along with a target classifier to evaluate aleatoric uncertainty (AU) based on class confusion. In the example selection phase, $kb$ examples with the lowest EU scores are chosen first, followed by querying $b$ examples with the highest AU scores, where $k$ is adaptively adjusted based on the target precision. In the Oracle labeling phase, the queried examples are assigned labels, and all relevant data pools are updated accordingly.
  • Figure 3: Test accuracy comparison on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
  • Figure 4: Query precision comparison on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
  • Figure 5: Ablation results on CIFAR-100 with a mismatch ratio of 40%. 1) The left validates the effectiveness of each method component. "Part 1" represents energy-based epistemic uncertainty and "Part 2" indicates energy-based aleatoric uncertainty. "mQP" denotes mean query precision across all AL rounds. 2) The middle assesses the sensitivity of the energy loss weight $\lambda_e$. "mDA" denotes mean detector test accuracy across all AL rounds. 3) The right shows the runtime comparison. The numbers on the bar chart correspond to mQP scores.
  • ...and 5 more figures

Theorems & Definitions (2)

  • Remark 1
  • Remark 2