Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nearest-Class Mean and Logits Agreement for Wildlife Open-Set Recognition

Jiahao Huo, Mufhumudzi Muthivhi, Terence L. van Zyl, Fredrik Gustafsson

TL;DR

This study contributes a post-processing OSR method that measures the agreement between the models' features and predicted logits, and proposes a probability distribution based on an input's distance to its Nearest Class Mean (NCM).

Abstract

Current state-of-the-art Wildlife classification models are trained under the closed world setting. When exposed to unknown classes, they remain overconfident in their predictions. Open-set Recognition (OSR) aims to classify known classes while rejecting unknown samples. Several OSR methods have been proposed to model the closed-set distribution by observing the feature, logit, or softmax probability space. A significant drawback of many existing approaches is the requirement to retrain the pre-trained classification model with the OSR-specific strategy. This study contributes a post-processing OSR method that measures the agreement between the models' features and predicted logits. We propose a probability distribution based on an input's distance to its Nearest Class Mean (NCM). The NCM-based distribution is then compared with the softmax probabilities from the logit space to measure agreement between the NCM and the classification head. Our proposed strategy ranks within the top three on two evaluated datasets, showing consistent performance across the two datasets. In contrast, current state-of-the-art methods excel on a single dataset. We achieve an AUROC of 93.41 and 95.35 for African and Swedish animals. The code can be found https://github.com/Applied-Representation-Learning-Lab/OSR.

Nearest-Class Mean and Logits Agreement for Wildlife Open-Set Recognition

TL;DR

This study contributes a post-processing OSR method that measures the agreement between the models' features and predicted logits, and proposes a probability distribution based on an input's distance to its Nearest Class Mean (NCM).

Abstract

Current state-of-the-art Wildlife classification models are trained under the closed world setting. When exposed to unknown classes, they remain overconfident in their predictions. Open-set Recognition (OSR) aims to classify known classes while rejecting unknown samples. Several OSR methods have been proposed to model the closed-set distribution by observing the feature, logit, or softmax probability space. A significant drawback of many existing approaches is the requirement to retrain the pre-trained classification model with the OSR-specific strategy. This study contributes a post-processing OSR method that measures the agreement between the models' features and predicted logits. We propose a probability distribution based on an input's distance to its Nearest Class Mean (NCM). The NCM-based distribution is then compared with the softmax probabilities from the logit space to measure agreement between the NCM and the classification head. Our proposed strategy ranks within the top three on two evaluated datasets, showing consistent performance across the two datasets. In contrast, current state-of-the-art methods excel on a single dataset. We achieve an AUROC of 93.41 and 95.35 for African and Swedish animals. The code can be found https://github.com/Applied-Representation-Learning-Lab/OSR.
Paper Structure (17 sections, 7 equations, 4 figures, 3 tables)

This paper contains 17 sections, 7 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Works
  4. Thresholding Methods
  5. Post-processing Methods
  6. Methodology
  7. Distance and Prediction Probability Distributions
  8. NCM Agreement
  9. Experimental Setup
  10. Dataset
  11. Metrics
  12. Results
  13. Open-Set Recognition
  14. Closed and open-set Accuracy
  15. Discussion
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Closed vs Open set AUROC performance of the different OSR methods across African and Swedish wildlife. Our NCMAgreement strategy (star) consistently produces a high open and closed set performance for both datasets. Other methods achieve optimal performance on one dataset.
  • Figure 2: Diagram of our NCMAgreement method, where $f$ denotes the backbone, $g$ denotes the two-layer classifier, and $u$ denotes the mean feature vector for each class. $p$ denotes the NCMAgreement function that takes in both the NCM and classifier scores to output the final probability value between known and unknown inputs.
  • Figure 3: Examples of images in each of the African, Swedish, and OSR datasets that we use.
  • Figure 4: The F1 score comparison with each of the methods.