SCOD: From Heuristics to Theory
Vojtech Franc, Jakub Paplham, Daniel Prusa
TL;DR
The paper studies Selective Classification in the presence of Out-of-Distribution data (SCOD) and derives a theoretically grounded optimal strategy that combines a Bayes classifier for in-distribution data with a stochastic linear selector operating in a 2D feature space defined by the conditional risk $r(x)$ and the OOD/ID likelihood ratio $g(x)$. It proves that relying solely on ID data cannot PAC-learn SCOD in a distribution-free setting and introduces POSCOD, a practical plugin method that learns the optimal SCOD strategy from both ID data and an unlabeled mixture of ID/OOD data using a corrected sigmoid model for estimating $g(x)$. Empirical results show that POSCOD consistently outperforms existing OOD detection methods and SIRC across benchmarks, validating the theoretical findings and highlighting the advantage of the linear 2D selector. The work advances SCOD by (i) narrowing the predictor search to structure-guided strategies, (ii) establishing non-learnability under ID-only data, and (iii) providing a scalable plugin estimator applicable with unlabeled mixtures, all with strong practical impact for reliable selective prediction in open-world settings.
Abstract
This paper addresses the problem of designing reliable prediction models that abstain from predictions when faced with uncertain or out-of-distribution samples - a recently proposed problem known as Selective Classification in the presence of Out-of-Distribution data (SCOD). We make three key contributions to SCOD. Firstly, we demonstrate that the optimal SCOD strategy involves a Bayes classifier for in-distribution (ID) data and a selector represented as a stochastic linear classifier in a 2D space, using i) the conditional risk of the ID classifier, and ii) the likelihood ratio of ID and out-of-distribution (OOD) data as input. This contrasts with suboptimal strategies from current OOD detection methods and the Softmax Information Retaining Combination (SIRC), specifically developed for SCOD. Secondly, we establish that in a distribution-free setting, the SCOD problem is not Probably Approximately Correct learnable when relying solely on an ID data sample. Third, we introduce POSCOD, a simple method for learning a plugin estimate of the optimal SCOD strategy from both an ID data sample and an unlabeled mixture of ID and OOD data. Our empirical results confirm the theoretical findings and demonstrate that our proposed method, POSCOD, out performs existing OOD methods in effectively addressing the SCOD problem.