Outperformance Score: A Universal Standardization Method for Confusion-Matrix-Based Classification Performance Metrics

TL;DR

This work tackles the challenge of comparing heterogeneous confusion-matrix-based classification metrics under varying class imbalance. It introduces the outperformance score (OPS), a universal standardization that maps any CMBCP metric to a [0,1] scale by evaluating percentile-rank relative to a prevalence-conditioned reference distribution. The framework separately handles labeling and scoring metrics, using analytical treatment for labeling metrics and a Monte Carlo approach with a Directed Binary Tree (DBT) distribution for scoring metrics. Through experiments on Heart Disease and Loan Default datasets across prediction, risk identification, and recommendation tasks, OPS demonstrates robust, interpretable cross-context comparisons and drift monitoring, without forcing practitioners to abandon their preferred metrics. The method promises broad applicability, including multi-label extensions and uncertainty quantification, to enhance transparency and comparability in applied ML evaluation.

Abstract

Many classification performance metrics exist, each suited to a specific application. However, these metrics often differ in scale and can exhibit varying sensitivity to class imbalance rates in the test set. As a result, it is difficult to use the nominal values of these metrics to interpret and evaluate classification performances, especially when imbalance rates vary. To address this problem, we introduce the outperformance score function, a universal standardization method for confusion-matrix-based classification performance (CMBCP) metrics. It maps any given metric to a common scale of $[0,1]$, while providing a clear and consistent interpretation. Specifically, the outperformance score represents the percentile rank of the observed classification performance within a reference distribution of possible performances. This unified framework enables meaningful comparison and monitoring of classification performance across test sets with differing imbalance rates. We illustrate how the outperformance scores can be applied to a variety of commonly used classification performance metrics and demonstrate the robustness of our method through experiments on real-world datasets spanning multiple classification applications.

Figures (7)

• Figure 1: Geometric representation of $\textnormal{OPS}_{f1}$ when (a) $\pi=0.1$; (b) $\pi=0.5$. And (c) plots the OPS function of f1_score given different $\pi$.
• Figure 2: A Directed Binary Tree distribution with depth = 3.
• Figure 3: The OPS functions for PRC with regard to: (a) AUC, and (b) the Precision corresponding to Recall=0.8, conditional on different imbalance rates.
• Figure 4: Visualize the XgBoost classifier's overall risk identification performance on the Heart Disease dataset using (a) the Precision-Recall curve (PRC), and (b) the OPS(Precision)-Recall curve (OPRC).
• Figure 5: Visualize the XgBoost classifier's overall risk identification performance on the Loan Default dataset using (a) the Precision-Recall curve (PRC), and (b) the OPS(Precision)-Recall curve (OPRC).

Theorems & Definitions (4)

• Example 1: Relative performance of f1_score under class imbalance
• Definition 1: Outperformance Score Function
• Example 2: Outperformance Score based on f1_score
• Example 3: Outperformance Score of PRC