Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProSub: Probabilistic Open-Set Semi-Supervised Learning with Subspace-Based Out-of-Distribution Detection

Erik Wallin, Lennart Svensson, Fredrik Kahl, Lars Hammarstrand

TL;DR

ProSub tackles open-set semi-supervised learning by introducing an ID subspace-based score, $s(oldsymbol{z})$, defined as the cosine of the angle between features and the ID subspace $W_ ext{id}$ spanned by class means. It then learns probabilistic predictions for ID/OOD by modeling $p_ ext{id}(s)$ and $p_ ext{ood}(s)$ as Beta distributions and estimating their parameters via a batch EM-like IMM procedure, enabling sampling-based ID/OOD decisions. A subspace loss $ ext{l}_{ ext{sub}}$ and a FixMatch-like pseudo-labeling scheme, augmented with cosine self-supervision, drive the model to separate ID from OOD while learning robust representations from all unlabeled data. Across multiple benchmarks, ProSub achieves state-of-the-art closed-set accuracy and AUROC for ID/OOD detection, demonstrating that probabilistic, subspace-based ID/OOD discrimination coupled with SSL signals yields strong OSSL performance; code is available at the provided repository.

Abstract

In open-set semi-supervised learning (OSSL), we consider unlabeled datasets that may contain unknown classes. Existing OSSL methods often use the softmax confidence for classifying data as in-distribution (ID) or out-of-distribution (OOD). Additionally, many works for OSSL rely on ad-hoc thresholds for ID/OOD classification, without considering the statistics of the problem. We propose a new score for ID/OOD classification based on angles in feature space between data and an ID subspace. Moreover, we propose an approach to estimate the conditional distributions of scores given ID or OOD data, enabling probabilistic predictions of data being ID or OOD. These components are put together in a framework for OSSL, termed ProSub, that is experimentally shown to reach SOTA performance on several benchmark problems. Our code is available at https://github.com/walline/prosub.

ProSub: Probabilistic Open-Set Semi-Supervised Learning with Subspace-Based Out-of-Distribution Detection

TL;DR

ProSub tackles open-set semi-supervised learning by introducing an ID subspace-based score, , defined as the cosine of the angle between features and the ID subspace spanned by class means. It then learns probabilistic predictions for ID/OOD by modeling and as Beta distributions and estimating their parameters via a batch EM-like IMM procedure, enabling sampling-based ID/OOD decisions. A subspace loss and a FixMatch-like pseudo-labeling scheme, augmented with cosine self-supervision, drive the model to separate ID from OOD while learning robust representations from all unlabeled data. Across multiple benchmarks, ProSub achieves state-of-the-art closed-set accuracy and AUROC for ID/OOD detection, demonstrating that probabilistic, subspace-based ID/OOD discrimination coupled with SSL signals yields strong OSSL performance; code is available at the provided repository.

Abstract

In open-set semi-supervised learning (OSSL), we consider unlabeled datasets that may contain unknown classes. Existing OSSL methods often use the softmax confidence for classifying data as in-distribution (ID) or out-of-distribution (OOD). Additionally, many works for OSSL rely on ad-hoc thresholds for ID/OOD classification, without considering the statistics of the problem. We propose a new score for ID/OOD classification based on angles in feature space between data and an ID subspace. Moreover, we propose an approach to estimate the conditional distributions of scores given ID or OOD data, enabling probabilistic predictions of data being ID or OOD. These components are put together in a framework for OSSL, termed ProSub, that is experimentally shown to reach SOTA performance on several benchmark problems. Our code is available at https://github.com/walline/prosub.
Paper Structure (31 sections, 14 equations, 12 figures, 10 tables, 2 algorithms)

This paper contains 31 sections, 14 equations, 12 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Model
  4. Proposing the Subspace Score
  5. Estimating a Probabilistic Model
  6. Enhancing OOD Detection with a Subspace Loss
  7. Pseudo-labeling
  8. Self-supervision
  9. Final Training Objective
  10. Optimization and Data Augmentation
  11. Experiments and Results
  12. Implementation Details
  13. Analyzing Density Estimation and $\ell_\text{sub}$
  14. Ablation: Self-supervision Enables the Subspace Score
  15. Ablation: Alternative Designs for the Subspace Score
  16. ...and 16 more sections

Figures (12)

  • Figure 1: Our ID subspace, $W_\text{id}$, spanned by the class centers. The angle, $\theta$, to this space is generally larger for OOD data than for ID data and is used as a score in ProSub.
  • Figure 2: Left: The flow of unlabeled data in ProSub. Right: Details for ID/OOD detection, the main contribution of ProSub.
  • Figure 3: Results from our estimation approach as specified in \ref{['alg:em-algorithm']} both from the warm-up stage and the subsequent training stage (with $\ell_\text{sub}$ applied).
  • Figure 3: AUROCs for alternatives to $s$.
  • Figure 4: Angles to $W_\text{id}$ vs. angles within $W_\text{id}$ to the closest class-mean for ID data.
  • ...and 7 more figures