A Framework for Optimizing Human-Machine Interaction in Classification Systems

TL;DR

This work addresses how to optimize human oversight in binary classification using a double-threshold policy, where scores below $\tau_l$ are auto-negative, above $\tau_u$ are auto-positive, and intermediate scores are sent to human reviewers. It formalizes the problem as maximizing the expected number of correct positives $C(\tau_l,\tau_u)$ under a fixed review budget, with explicit expressions for $FP$, $FN$, $TN$, and $H$, and analyzes how score distributions influence threshold settings and performance frontiers. Through Monte Carlo simulations over calibrated probability distributions, the study reveals how different distributions shape the Pareto frontier between accuracy metrics (e.g., F1) and human workload, and it demonstrates diminishing returns in performance as review budgets increase. The framework is broadly applicable across domains requiring selective human validation and provides a reproducible, data-driven method for planning automation versus human review in decision pipelines. Key technical insights include treatment of calibrated probabilities, the impact of distributional shape (e.g., Beta mixtures and skewness) on optimal thresholds, and the construction of Pareto frontiers to guide budget-to-performance decisions, with practical implications for domains from entity resolution to content moderation.

Abstract

Automated decision systems increasingly rely on human oversight to ensure accuracy in uncertain cases. This paper presents a practical framework for optimizing such human-in-the-loop classification systems using a double-threshold policy. Instead of relying on a single decision cutoff, the system defines two thresholds (a lower and an upper) to automatically accept or reject confident cases while routing ambiguous ones for human review. We formalize this problem as an optimization task that balances system accuracy against human review workload and demonstrate its behavior through extensive Monte Carlo simulations. Our results quantify how different probability score distributions affect the efficiency of human intervention and identify the regions of diminishing returns where additional review yields minimal benefit. The framework provides a general, reproducible method for improving reliability in any decision pipeline requiring selective human validation, including applications in entity resolution, fraud detection, medical triage, and content moderation.

Figures (13)

• Figure 1: Density plots of the simulated probability distributions used in the Monte Carlo experiments. The Beta mixture shows balanced bimodality, while the Beta Right Skewed and Beta Left Skewed distributions are skewed toward high and low probabilities, respectively.
• Figure 2: Expected true positives (%) as a function of the lower ($\tau_{l}$) and upper ($\tau_{u}$) thresholds across three simulated score distributions: Beta Mixture, Beta Right Skewed, and Beta Left Skewed. Each panel visualizes the expected percentage of correctly classified pairs for a unique $(\tau_{l}, \tau_{u})$ operating point. Color scales are panel-specific.
• Figure 3: Expected F1 as a function of the lower ($\tau_l$) and upper ($\tau_u$) thresholds under three simulated score distributions. Each panel visualizes the expected F1 score for a unique $(\tau_{l}, \tau_{u})$ operating point. Color scales are panel-specific.
• Figure 4: Pareto frontier of F1 vs. human review load. Each gray dot is one operating point defined by a pair of thresholds $(\tau_l,\tau_u)$. The outlined curve traces the Pareto frontier: the optimal settings that achieve the highest F1 for a given review budget. Example operating points are annotated with their $(\tau_l,\tau_u)$ values. The analysis is conducted under the Beta–mixture score regime.
• Figure 5: Precision and recall trade-offs under varying human review budgets. Each gray point represents an operating policy $(\tau_l,\tau_u)$, while the black-outlined curve marks the Pareto-optimal frontier. Annotated points indicate representative threshold pairs.