Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection

Rodrigo F. L. Lassance, Jasper De Bock

Abstract

Among the different possible strategies for evaluating the reliability of individual predictions of classifiers, robustness quantification stands out as a method that evaluates how much uncertainty a classifier could cope with before changing its prediction. However, its applicability is more limited than some of its alternatives, since it requires the use of generative models and restricts the analyses either to specific model architectures or discrete features. In this work, we propose a new robustness metric applicable to any probabilistic discriminative classifier and any type of features. We demonstrate that this new metric is capable of distinguishing between reliable and unreliable predictions, and use this observation to develop new strategies for dynamic classifier selection.

Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection

Abstract

Among the different possible strategies for evaluating the reliability of individual predictions of classifiers, robustness quantification stands out as a method that evaluates how much uncertainty a classifier could cope with before changing its prediction. However, its applicability is more limited than some of its alternatives, since it requires the use of generative models and restricts the analyses either to specific model architectures or discrete features. In this work, we propose a new robustness metric applicable to any probabilistic discriminative classifier and any type of features. We demonstrate that this new metric is capable of distinguishing between reliable and unreliable predictions, and use this observation to develop new strategies for dynamic classifier selection.
Paper Structure (10 sections, 1 theorem, 15 equations, 4 figures, 5 tables)

This paper contains 10 sections, 1 theorem, 15 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Setup
  3. Experiments
  4. Correlation with Accuracy
  5. Comparison Between Robustness Metrics
  6. Comparison Between Models
  7. Dynamic Selection of Classifiers
  8. Performance Under Label Corruption
  9. Discussion
  10. Proof of Theorem 1

Key Result

Theorem 1

Consider a measure $P$ in $\mathcal{P}(\mathcal{Y},\mathcal{X})$, its corresponding density $p$ and, for any given features $x\in\mathcal{X}$, the most and second most likely class according to $p$ given $x$: Then the robustness of the prediction $\hat{y}_1$ w.r.t. $d^*_{\mathrm{COR}}$ and $d_{\mathrm{COR}}$ is given by, respectively,

Figures (4)

  • Figure 1: Comparison between the ARCs of $r_{GeF}$ and $r_{d_{COR}}$ for two datasets.
  • Figure 2: Comparison between the ARCs of different models for two datasets when ordered by $r_{d_{COR}}$.
  • Figure 3: Robustness-based strategies for DS. $M_1$ is the model with the best accuracy for the validation set, while $M_2$ is the second best.
  • Figure 4: Boxplot of accuracies for each strategy and different datasets; white triangles represent the mean.

Theorems & Definitions (1)

  • Theorem 1