OTLP: Output Thresholding Using Mixed Integer Linear Programming

TL;DR

The paper addresses the problem of selecting optimal inference-time thresholds for probability-based classifiers, particularly in imbalanced settings. It introduces OTLP, a model-agnostic framework that uses mixed-integer linear programming to search for a single threshold (or threshold per subspace) by optimizing a user-defined objective under customizable constraints, based on validation data. Through extensive experiments on a Credit Card Fraud Detection dataset across multiple settings, OTLP demonstrates thresholds that outperform the default threshold and adapt to complex problem structures, including global/local and subspace constraints. The work provides a principled, constraint-aware thresholding approach that can be integrated into standard ML Pipelines, with potential for scalable extensions in future work.

Abstract

Output thresholding is the technique to search for the best threshold to be used during inference for any classifiers that can produce probability estimates on train and testing datasets. It is particularly useful in high imbalance classification problems where the default threshold is not able to refer to imbalance in class distributions and fail to give the best performance. This paper proposes OTLP, a thresholding framework using mixed integer linear programming which is model agnostic, can support different objective functions and different set of constraints for a diverse set of problems including both balanced and imbalanced classification problems. It is particularly useful in real world applications where the theoretical thresholding techniques are not able to address to product related requirements and complexity of the applications which utilize machine learning models. Through the use of Credit Card Fraud Detection Dataset, we evaluate the usefulness of the framework.

Figures (8)

• Figure 1: (a) shows a curve of mislassification cost with increasing threshold values in range $[0,1]$ whereas (b) shows a curve of Cohen's kappa metric with increasing threshold values in range. The objective function and related metrics is subject to change according to the characteristics of the dataset and the problem at hand.
• Figure 2: shows in which step the threshold optimization framework is run with respect to a classical machine learning model training flow. A machine learning model is trained in step (a). Sensitivity is generated on validation dataset using the trained model in step (a) which is further used by OTLP to select the optimized threshold in step (b). The threshold is later used to assign classes to the instances in the test dataset in step (c).
• Figure 3: OTLP selects the optimized threshold as the global maxima of the objective function within the specified constraint list.
• Figure 4: OTLP selects the optimized threshold as the global maxima of the objective function within the specified constraint list in the validation dataset. The train dataset used to train the model was balanced with 1:1 class ratio.
• Figure 5: OTLP selects the optimized threshold as the global maxima of the objective function within the specified constraint list in the validation dataset.