Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Building a stable classifier with the inflated argmax

Jake A. Soloff, Rina Foygel Barber, Rebecca Willett

TL;DR

The paper tackles the instability of multiclass classifiers that output a single label by focusing on stability of the final decision rather than predicted probabilities. It introduces a two-stage pipeline that stabilizes probability scores via bagging and uses a new inflated argmax to produce a set of candidate labels with provable selection stability, independent of data distribution, the number of classes $L$, or the covariate dimensionality. A key contribution is the definition of selection stability and the demonstration that bagging combined with the $\varepsilon$-compatible inflated argmax yields a concrete, distribution-free stability guarantee with an explicit bound on the instability parameter $\delta$. The framework is designed to keep predictions informative (small sets when confident) while ensuring robust performance under data perturbations, demonstrated experimentally on Fashion-MNIST where stability improves dramatically with negligible accuracy loss. These results offer a practical, scalable route to trustworthy multiclass classification in settings where stability and interpretability of the final label are crucial.

Abstract

We propose a new framework for algorithmic stability in the context of multiclass classification. In practice, classification algorithms often operate by first assigning a continuous score (for instance, an estimated probability) to each possible label, then taking the maximizer -- i.e., selecting the class that has the highest score. A drawback of this type of approach is that it is inherently unstable, meaning that it is very sensitive to slight perturbations of the training data, since taking the maximizer is discontinuous. Motivated by this challenge, we propose a pipeline for constructing stable classifiers from data, using bagging (i.e., resampling and averaging) to produce stable continuous scores, and then using a stable relaxation of argmax, which we call the "inflated argmax," to convert these scores to a set of candidate labels. The resulting stability guarantee places no distributional assumptions on the data, does not depend on the number of classes or dimensionality of the covariates, and holds for any base classifier. Using a common benchmark data set, we demonstrate that the inflated argmax provides necessary protection against unstable classifiers, without loss of accuracy.

Building a stable classifier with the inflated argmax

TL;DR

The paper tackles the instability of multiclass classifiers that output a single label by focusing on stability of the final decision rather than predicted probabilities. It introduces a two-stage pipeline that stabilizes probability scores via bagging and uses a new inflated argmax to produce a set of candidate labels with provable selection stability, independent of data distribution, the number of classes , or the covariate dimensionality. A key contribution is the definition of selection stability and the demonstration that bagging combined with the -compatible inflated argmax yields a concrete, distribution-free stability guarantee with an explicit bound on the instability parameter . The framework is designed to keep predictions informative (small sets when confident) while ensuring robust performance under data perturbations, demonstrated experimentally on Fashion-MNIST where stability improves dramatically with negligible accuracy loss. These results offer a practical, scalable route to trustworthy multiclass classification in settings where stability and interpretability of the final label are crucial.

Abstract

We propose a new framework for algorithmic stability in the context of multiclass classification. In practice, classification algorithms often operate by first assigning a continuous score (for instance, an estimated probability) to each possible label, then taking the maximizer -- i.e., selecting the class that has the highest score. A drawback of this type of approach is that it is inherently unstable, meaning that it is very sensitive to slight perturbations of the training data, since taking the maximizer is discontinuous. Motivated by this challenge, we propose a pipeline for constructing stable classifiers from data, using bagging (i.e., resampling and averaging) to produce stable continuous scores, and then using a stable relaxation of argmax, which we call the "inflated argmax," to convert these scores to a set of candidate labels. The resulting stability guarantee places no distributional assumptions on the data, does not depend on the number of classes or dimensionality of the covariates, and holds for any base classifier. Using a common benchmark data set, we demonstrate that the inflated argmax provides necessary protection against unstable classifiers, without loss of accuracy.
Paper Structure (47 sections, 15 theorems, 86 equations, 4 figures, 2 tables)

This paper contains 47 sections, 15 theorems, 86 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem setting
  3. Overview of main results
  4. Framework: stable classification
  5. Interpreting the definition
  6. Connection to classical algorithmic stability
  7. From stability to selection stability
  8. Methodology: assumption-free stable classification
  9. Bagging classifier weights
  10. Computational cost of bagging.
  11. The inflated argmax
  12. Inflated argmax leads to selection stability
  13. Proof of \ref{['thm:main']}.
  14. Additional properties of the inflated argmax
  15. Experiments
  16. ...and 32 more sections

Key Result

Proposition 5

Let $\mathcal{A}$ be a learning algorithm with tail stability $(\varepsilon,\delta)$ at sample size $n$, and let $s$ be a selection rule satisfying $\varepsilon$-compatibility. Then the classification algorithm $\mathcal{C}=s\circ\mathcal{A}$ has selection stability $\delta$ at sample size $n$.

Figures (4)

  • Figure 1: The left plot illustrates the inflated argmax \ref{['eqn:define-stable-argmax']} over the simplex $\Delta_{L-1}$ when $L=3$. The numbers in brackets correspond to the output of the inflated argmax, $\mathop{\mathrm{argmax}}\nolimits^\varepsilon(w)$, for various points $w$ in the simplex. The right plot shows the same but for the standard argmax, which corresponds to the limit of $\mathop{\mathrm{argmax}}\nolimits^\varepsilon(w)$ as $\varepsilon \rightarrow 0$.
  • Figure 2: Results on the Fashion MNIST data set. The figure shows the instability $\delta_j$ (defined in \ref{['eqn:MNIST_delta_j']}) over all test points $j=1,\dots,N$. The curves display the fraction of $\delta_j$'s that exceed $\delta$, for each value $\delta\in[0,1]$. The vertical axis is on a log scale. See \ref{['sec:experiments']} for details.
  • Figure 3: The fixed-margin selection rule \ref{['eqn:selection-rule-margin']}, left, and the inflated argmax \ref{['eqn:define-stable-argmax']}, right, when $L=3$.
  • Figure 4: Simulation to compare the selection rules $\mathop{\mathrm{argmax}}\nolimits^\varepsilon$ and $s_{\textnormal{margin}}^\varepsilon$. The figure shows the average set size, $|\mathop{\mathrm{argmax}}\nolimits^\varepsilon(w)|$ and $|s_{\textnormal{margin}}^\varepsilon(w)|$, averaged over $1,000$ random draws of $w$ (with standard error bars shown). See \ref{['sec:compare_sargmax_to_margin_sim']} for details.

Theorems & Definitions (20)

  • Definition 1: Inflated argmax
  • Definition 2: Selection stability
  • Definition 3: Tail stability
  • Definition 4: $\varepsilon$-compatibility
  • Proposition 5
  • Theorem 6
  • Definition 7: Bootstrapping or subbagging a base learning algorithm $\mathcal{A}$
  • Theorem 8: soloff2024stability
  • Theorem 9
  • Proposition 10: Basic properties of the inflated argmax
  • ...and 10 more