Machine Learning with a Reject Option: A survey

TL;DR

This survey addresses the problem of machine learning with a reject option, focusing on how and when models should abstain to avoid costly mispredictions. It introduces three architectural families (separated, dependent, integrated) for operationalizing abstention, formalizes two rejection types (ambiguity and novelty), and reviews evaluation schemes (fixed-rate, trade-off curves, and cost-based). The paper also surveys learning strategies for each architecture, strategies for combining multiple rejectors, and highlights applications in biomedicine, engineering, economics, and image analysis, while linking rejection to uncertainty quantification, anomaly detection, active learning, open-world learning, delegation, and meta-learning. Finally, it outlines research questions and future directions, including standard benchmarks, partial rejection, and extending rejection techniques beyond classification. The practical impact is to provide a structured framework for building trustworthy, cost-aware systems that can safely defer uncertain decisions to human experts or alternative processes.

Abstract

Machine learning models always make a prediction, even when it is likely to be inaccurate. This behavior should be avoided in many decision support applications, where mistakes can have severe consequences. Albeit already studied in 1970, machine learning with rejection recently gained interest. This machine learning subfield enables machine learning models to abstain from making a prediction when likely to make a mistake. This survey aims to provide an overview on machine learning with rejection. We introduce the conditions leading to two types of rejection, ambiguity and novelty rejection, which we carefully formalize. Moreover, we review and categorize strategies to evaluate a model's predictive and rejective quality. Additionally, we define the existing architectures for models with rejection and describe the standard techniques for learning such models. Finally, we provide examples of relevant application domains and show how machine learning with rejection relates to other machine learning research areas.

Figures (8)

• Figure 1: Data flow in a separated rejector, in which both the predictor and the rejector are stand-alone models. The rejector is a filter and only passes accepted examples to the predictor.
• Figure 2: Data flow in a dependent rejector. First, the predictor processes the example at hand. Next, the rejector assesses the confidence in the prediction based on the predictor's representation of the example.
• Figure 3: Data flow in an integrated rejector, in which the predictor and rejector are one model. This model embeds the reject and predict functions and directly outputs a prediction or a rejection.
• Figure 4: Illustration of a classification scenario where the two classes overlap in a region. The dotted lines represent the rejector, the dash-dotted line the fitted predictor $h$, and the solid line the ground truth relation $f$. a) shows ambiguity rejection due to a non-deterministic relation between $X$ and $Y$, b) introduces ambiguity rejection due to the model bias, and c) illustrates an example of novelty rejection. While in the first two plots the rejection region is inside the two dotted lines (examples with cross marks are rejected), in the third figure the rejected novel examples (stars) are outside the dotted line.
• Figure 5: Illustration of a regression scenario where rejection can be applied. The dash-dotted line represents the predictor $h$, while the solid line indicates the true function $f$. In a), examples in between the dotted lines are rejected (cross mark) due to the high variance of $Y$. In b), examples indicated with the cross mark are rejected due to the incorrect model bias. In c), star-marked examples are rejected because they differ from the training data (novelties).