Cost-Sensitive Evaluation for Binary Classifiers
Pierangelo Lombardo, Antonio Casoli, Cristian Cingolani, Shola Oshodi, Michele Zanatta
TL;DR
This work tackles the challenge of evaluating binary classifiers under cost considerations by introducing Weighted Accuracy (WA), a simple, interpretable metric that aligns with minimizing the Total Classification Cost (TCC) when unit costs are known. It develops a general cost-sensitive evaluation framework that accommodates class imbalance without resampling, extends WA to target datasets with unequal UCCs, and proposes methods to estimate WA weights directly from data or via emblematic-model rankings. The paper demonstrates, through example-dependent costs (e.g., churn prediction), that WA and related metrics (e.g., H-informed, EWA) correlate robustly with TCC across diverse scenarios, while traditional rebalancing metrics can mislead under certain cost and distribution conditions. Practically, WA enables cross-dataset comparisons and ROI-oriented classifier optimization, providing a principled, scalable alternative for cost-efficient decision making in imbalanced binary classification tasks.
Abstract
Selecting an appropriate evaluation metric for classifiers is crucial for model comparison and parameter optimization, yet there is not consensus on a universally accepted metric that serves as a definitive standard. Moreover, there is often a misconception about the perceived need to mitigate imbalance in datasets used to train classification models. Since the final goal in classifier optimization is typically maximizing the return of investment or, equivalently, minimizing the Total Classification Cost (TCC), we define Weighted Accuracy (WA), an evaluation metric for binary classifiers with a straightforward interpretation as a weighted version of the well-known accuracy metric, coherent with the need of minimizing TCC. We clarify the conceptual framework for handling class imbalance in cost-sensitive scenarios, providing an alternative to rebalancing techniques. This framework can be applied to any metric that, like WA, can be expressed as a linear combination of example-dependent quantities and allows for comparing the results obtained in different datasets and for addressing discrepancies between the development dataset, used to train and validate the model, and the target dataset, where the model will be deployed. It also specifies in which scenarios using UCCs-unaware class rebalancing techniques or rebalancing metrics aligns with TCC minimization and when it is instead counterproductive. Finally, we propose a procedure to estimate the WA weight parameter in the absence of fully specified UCCs and demonstrate the robustness of WA by analyzing its correlation with TCC in example-dependent scenarios.