A Connection Between Learning to Reject and Bhattacharyya Divergences
Alexander Soen
TL;DR
This work studies learning to reject in classification by linking abstention to divergence-based thresholding. It moves from a marginal ideal-distribution framework to a joint ideal distribution over inputs and labels, deriving a density-ratio rejector that can be thresholded to abstain. When using the canonical KL divergence, the marginal approach recovers Chow's Rule, while a joint formulation with a modified log-loss yields rejection rules based on skewed Bhattacharyya divergences, offering a less aggressive rejection regime. The results unify rejection with information-theoretic divergences and provide principled alternatives for abstention in practical settings, with potential implications for calibration and deferral in complex decision systems.
Abstract
Learning to reject provide a learning paradigm which allows for our models to abstain from making predictions. One way to learn the rejector is to learn an ideal marginal distribution (w.r.t. the input domain) - which characterizes a hypothetical best marginal distribution - and compares it to the true marginal distribution via a density ratio. In this paper, we consider learning a joint ideal distribution over both inputs and labels; and develop a link between rejection and thresholding different statistical divergences. We further find that when one considers a variant of the log-loss, the rejector obtained by considering the joint ideal distribution corresponds to the thresholding of the skewed Bhattacharyya divergence between class-probabilities. This is in contrast to the marginal case - that is equivalent to a typical characterization of optimal rejection, Chow's Rule - which corresponds to a thresholding of the Kullback-Leibler divergence. In general, we find that rejecting via a Bhattacharyya divergence is less aggressive than Chow's Rule.