Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AHA: Human-Assisted Out-of-Distribution Generalization and Detection

Haoyue Bai, Jifan Zhang, Robert Nowak

TL;DR

A novel, integrated approach AHA (Adaptive Human-Assisted OOD learning) to simultaneously address both OOD generalization and detection through a human-assisted framework by labeling data in the wild by strategically labels examples within a novel maximum disambiguation region.

Abstract

Modern machine learning models deployed often encounter distribution shifts in real-world applications, manifesting as covariate or semantic out-of-distribution (OOD) shifts. These shifts give rise to challenges in OOD generalization and OOD detection. This paper introduces a novel, integrated approach AHA (Adaptive Human-Assisted OOD learning) to simultaneously address both OOD generalization and detection through a human-assisted framework by labeling data in the wild. Our approach strategically labels examples within a novel maximum disambiguation region, where the number of semantic and covariate OOD data roughly equalizes. By labeling within this region, we can maximally disambiguate the two types of OOD data, thereby maximizing the utility of the fixed labeling budget. Our algorithm first utilizes a noisy binary search algorithm that identifies the maximal disambiguation region with high probability. The algorithm then continues with annotating inside the identified labeling region, reaping the full benefit of human feedback. Extensive experiments validate the efficacy of our framework. We observed that with only a few hundred human annotations, our method significantly outperforms existing state-of-the-art methods that do not involve human assistance, in both OOD generalization and OOD detection. Code is publicly available at \url{https://github.com/HaoyueBaiZJU/aha}.

AHA: Human-Assisted Out-of-Distribution Generalization and Detection

TL;DR

A novel, integrated approach AHA (Adaptive Human-Assisted OOD learning) to simultaneously address both OOD generalization and detection through a human-assisted framework by labeling data in the wild by strategically labels examples within a novel maximum disambiguation region.

Abstract

Modern machine learning models deployed often encounter distribution shifts in real-world applications, manifesting as covariate or semantic out-of-distribution (OOD) shifts. These shifts give rise to challenges in OOD generalization and OOD detection. This paper introduces a novel, integrated approach AHA (Adaptive Human-Assisted OOD learning) to simultaneously address both OOD generalization and detection through a human-assisted framework by labeling data in the wild. Our approach strategically labels examples within a novel maximum disambiguation region, where the number of semantic and covariate OOD data roughly equalizes. By labeling within this region, we can maximally disambiguate the two types of OOD data, thereby maximizing the utility of the fixed labeling budget. Our algorithm first utilizes a noisy binary search algorithm that identifies the maximal disambiguation region with high probability. The algorithm then continues with annotating inside the identified labeling region, reaping the full benefit of human feedback. Extensive experiments validate the efficacy of our framework. We observed that with only a few hundred human annotations, our method significantly outperforms existing state-of-the-art methods that do not involve human assistance, in both OOD generalization and OOD detection. Code is publicly available at \url{https://github.com/HaoyueBaiZJU/aha}.

Paper Structure

This paper contains 27 sections, 6 equations, 3 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Setup
  4. Methodology
  5. Baseline Labeling Regions
  6. Maximum Disambiguation Region
  7. Algorithm
  8. Learning Objective
  9. Experiments
  10. Experiment Setup
  11. Main Results and Discussion
  12. Ablation Studies
  13. Qualitative Analysis
  14. Conclusion
  15. ConfUpdate: Shrinking Confidence Interval for Labeling
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Illustration and comparison of three different labeling regions. The horizontal axis is the OOD score, and the vertical axis is the frequency. Note that we color the three different sub-distributions (ID, covariate OOD, semantic OOD) separately for clarity. In practice, the membership is not revealed on these unlabeled wild data.
  • Figure 2: (a)-(b): T-SNE visualization of the image embeddings for ERM vs. AHA (ours). (c)-(d) Score distributions for ERM vs. AHA (ours). Different colors represent the different types of test data: CIFAR-10 as $\mathbb{P}_{\text{in}}$ (blue), CIFAR-10-C as $\mathbb{P}_\text{out}^\text{covariate}$ (green), and Textures as $\mathbb{P}_\text{out}^\text{semantic}$ (gray).
  • Figure 3: (a)-(e) Score distributions for the real wild data. Different colors represent the different types of test data: CIFAR-10 as $\mathbb{P}_{\text{in}}$ (blue), CIFAR-10-C as $\mathbb{P}_\text{out}^\text{covariate}$ (green), and Textures as $\mathbb{P}_\text{out}^\text{semantic}$ (gray).

Theorems & Definitions (1)

  • Definition 1: Maximum Ambiguity Threshold