Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints

TL;DR

This paper addresses the challenge of evaluating binary classifiers when deployment imposes asymmetric costs and operational constraints. It extends the existing VOROS metric by defining a feasible ROC region under a minimum precision bound $\alpha$ and a maximum capacity bound $\kappa$, and then derives computable partial-area and partial-volume measures $A_t^*(\cdot)$ and $\text{PV}(\cdot)$ that summarize cost-aware performance across a range of the fractional cost $t$. The approach yields 1) a geometric characterization of feasible ROC regions, 2) tractable formulas for partial areas and volumes, and 3) empirical demonstrations in hospital-alert contexts (MIMIC-IV mortality prediction and EW scores) showing improved classifier ranking under realistic constraints. The results indicate that partial VOROS provides a more operation-ready ranking than unconstrained VOROS or AUROC-like metrics, enabling deployment-relevant model selection and thresholding. The method’s explicit handling of precision and capacity constraints makes it particularly suitable for high-stakes alert systems where false alarms and staff workload are critical considerations.

Abstract

The ROC curve is widely used to assess binary classification performance. Yet for some applications such as alert systems for hospitalized patient monitoring, conventional ROC analysis cannot capture crucial factors that impact deployment, such as enforcing a minimum precision constraint to avoid false alarm fatigue or imposing an upper bound on the number of predicted positives to represent the capacity of hospital staff. The usual area under the curve metric also does not reflect asymmetric costs for false positives and false negatives. In this paper we address all three of these issues. First, we show how the subset of classifiers that meet given precision and capacity constraints can be represented as a feasible region in ROC space. We establish the geometry of this feasible region. We then define the partial area of lesser classifiers, a performance metric that is monotonic with cost and only accounts for the feasible portion of ROC space. Averaging this area over a desired range of cost parameters results in the partial volume over the ROC surface, or partial VOROS. In experiments predicting mortality risk using vital sign history on the MIMIC-IV dataset, we show this cost-aware metric is better than alternatives for ranking classifiers in hospital alert applications.

Figures (5)

• Figure 1: Overview of VOROS (left) and our new partial VOROS (right). Left: At each fractional cost parameter $t \in [0, 1]$, we draw iso-performance lines for points $P$ and $Q$ in ROC space. Solid lines have lower cost than dashed lines. Points below each line have higher cost than that line. A point's area of lesser classifiers, colored light pink here for point $P$, is the area below that point's iso-performance line. The VOROS ratigan2024voros for a classifier is computed by finding at each $t$ the maximum area of lesser classifiers for any point in its ROC curve, then integrating over a desired range of $t$. Right: Our partial VOROS excludes regions in gray that do not achieve a minimum precision $\alpha$ or exceed a maximum capacity $\kappa$ (yield too many positive predictions). These limits correspond to linear constraints in ROC space.
• Figure 2: Cases for distinct polygons of feasible region (colored in gold), assuming the practical facts in Def. \ref{['def:practical_assumptions']}. Created using $|\mathcal{P}|=1000$, $|\mathcal{N}|=9000$, so $p = 0.1$, roughly matching the mortality alert application on MIMIC-IV in Sec. \ref{['sec:mimiciv_experiments']}.
• Figure 3: Formula for partial area of lesser classifiers for a given $(h,k)$ location in ROC space and cost parameter $t \in [0, 1]$. The 4 cases here depend on the geometry of the feasible region and the iso-performance line $\ell_t$ through $(h,k)$. The appendix identifies which case is needed for given inputs. The cases above are, in order, the areas of $v_{00}v_{\alpha t}v_{0t}$, $v_{00}v_{\alpha\kappa}v_{\kappa t}v_{0t}$, $v_{00}v_{\alpha \kappa}v_{\kappa 1}v_{t1}v_{0t}$, and $v_{00}v_{\alpha 1}v_{t1}v_{0t}$ respectively, using the vertex definitions in Def. \ref{['def:vertices']}.
• Figure 4: Re-analysis of Early Warning Scores for Deterioration, using ROC curves from edelsonEarlyWarningScores2024. Top: ROC curves with strict (-- lines) and more lenient (:) constraints. Bottom: Heatmap of models with superior partial VOROS scores over possible $\alpha, \kappa$ limits.
• Figure 5: Developing mortality alert systems on MIMIC-IV. We compare our partial VOROS against other strategies for selecting model family, hyperparameters, and decision thresholds using validation data. We evaluate each selection's binary alerts on test data in terms of average cost over the provided $C_0/C_1$ distribution. Left: The ROC curve on heldout data in the feasible region (gold), for the two best selection strategies in scenario 1. Center: The maximum feasible area of lesser classifiers across a range of cost parameter values $t$, with the corresponding cost ratio $C_0/C_1$ (false positive to false negative) on the top axis. Right: Table of costs on test set for each strategy and scenario.

Theorems & Definitions (38)

• Definition 1: ROC Space
• Definition 2: Fractional Cost Parameter
• Definition 3: Cost
• Definition 4: ROC Surface
• Definition 5: Iso-performance line
• Definition 6: Precision
• Definition 7: Precision Bound
• Lemma 8: Precision Bound in ROC Space
• proof
• Definition 9: Capacity bound