Ask for More Than Bayes Optimal: A Theory of Indecisions for Classification

TL;DR

This work develops a principled theory of selective classification that achieves user-specified accuracy by abstaining on indecisions, quantified through the minimax risk ${\mathcal{R}}(\gamma)$ and the optimal rule $Y_{\gamma}^*$. It connects selective classification to Neyman–Pearson testing, providing finite-sample calibration methods for both classification and hypothesis testing, with rigorous guarantees. A key theoretical contribution is a sharp phase transition in the Gaussian mixture setting, showing near-Bayes accuracy can be obtained with vanishing indecision mass, and the framework is extended to plug-in rules, MLR-adaptations, and multi-class problems. Empirical results on Gaussian mixtures and real data (COMPAS) confirm that even small indecision masses yield meaningful accuracy gains while maintaining risk control, making indecision a practical tool for high-stakes risk management.

Abstract

Selective classification is a powerful tool for automated decision-making in high-risk scenarios, allowing classifiers to act only when confident and abstain when uncertainty is high. Given a target accuracy, our goal is to minimize indecisions, observations we do not automate. For difficult problems, the target accuracy may be unattainable without abstention. By using indecisions, we can control the misclassification rate to any user-specified level, even below the Bayes optimal error rate, while minimizing overall indecision mass. We provide a complete characterization of the minimax risk in selective classification, establishing continuity and monotonicity properties that enable optimal indecision selection. We revisit selective inference via the Neyman-Pearson testing framework, where indecision enables control of type 2 error given fixed type 1 error probability. For both classification and testing, we propose a finite-sample calibration method with non-asymptotic guarantees, proving plug-in classifiers remain consistent and that accuracy-based calibration effectively controls indecision mass. In the binary Gaussian mixture model, we uncover the first sharp phase transition in selective inference, showing minimal indecision can yield near-optimal accuracy even under poor class separation. Experiments on Gaussian mixtures and real datasets confirm that small indecision proportions yield substantial accuracy gains, making indecision a principled tool for risk control.

Figures (7)

• Figure 1: An example of a classification scenario where the data comes from two different normal distributions. Low Risk $\sim N(0,1)$ and High Risk $\sim N(2,1)$. Left plot: Classification with no indecisions. Right Plot: Classification with indecisions (highlighted in yellow). The indecisions do not contribute to the risk of our classifier. By including the indecisions, we are able to obtain a much lower specified level of control over the risk.
• Figure 2: The best level of accuracy that can be obtained by the bayes classifier and classification with indecisions as the distance between the underlying data distributions gets further apart. $\Delta$ represents the amount of separation between the High and Low risk classes. A larger $\Delta$ means that the classification problem is easier.
• Figure 3: An example of a binary classification problem that includes indecisions (orange / open circle). For the left most figure, the indecisions lie in a region between the two classes: class 1 (green / solid circle) and class 2 (blue / square). A plateau at the threshold $\tau_\gamma$ indicates that some observations may be randomly classified as either class 1 or an indecision. In the right most figure, the indecisions lie below the threshold $\tau_\gamma$ in comparison to the largest conditional density across potentially many classes. This demonstrates that the indecision region may not be a simple interval.
• Figure 4: A comparison of Neyman-Pearson (NP) Classification (left) and Selective Classification (right), which can use indecisions. The NP-classifier is able to control the type 1 error at the correct level, at the compromise of the type 2 error. In contrast, selective classification is able to control both the type 1 and type 2 errors at the correct level, through the introduction of indecisions (yellow shaded region).
• Figure 5: Computation of $\mathcal{R}(\gamma)/\delta$ with $\delta=10^{-7}$ (left) and $\delta=10^{-15}$ (right). The lower bound corresponds to the curve $m_*(c)$ while the upper bound corresponds to $m^*(c)$.

Theorems & Definitions (20)

• Theorem 1
• Proposition 1
• Lemma 1
• proof
• Theorem 2
• Proposition 2
• Definition 1
• Theorem 3
• Lemma 2
• Remark 1