Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings

TL;DR

The paper proposes the suitability filter, a modular statistical framework for detecting degradation in classifier performance on unlabeled deployment data by comparing the estimated accuracy on $D_u$ to the labeled accuracy on $D_{ ext{test}}$ within a margin $m$ using non-inferiority testing. It combines a per-sample correctness estimator $C$, generated from domain-agnostic suitability signals, with calibration considerations and a Welch's $t$-test to yield a dataset-level SUITABLE/INCONCLUSIVE decision. The approach is grounded in theoretical guarantees for false-positive control under $\delta$-calibration, and is empirically validated on multiple WILDS benchmarks (FMoW-WILDS, RxRx1-WILDS, CivilComments-WILDS) across various covariate shifts, demonstrating reliable detection of deterioration and practical deployment insights. The work aims to enable auditable SLAs and safer, more trustworthy deployment by offering a label-free mechanism to monitor model performance in dynamic environments.

Abstract

Deploying machine learning models in safety-critical domains poses a key challenge: ensuring reliable model performance on downstream user data without access to ground truth labels for direct validation. We propose the suitability filter, a novel framework designed to detect performance deterioration by utilizing suitability signals -- model output features that are sensitive to covariate shifts and indicative of potential prediction errors. The suitability filter evaluates whether classifier accuracy on unlabeled user data shows significant degradation compared to the accuracy measured on the labeled test dataset. Specifically, it ensures that this degradation does not exceed a pre-specified margin, which represents the maximum acceptable drop in accuracy. To achieve reliable performance evaluation, we aggregate suitability signals for both test and user data and compare these empirical distributions using statistical hypothesis testing, thus providing insights into decision uncertainty. Our modular method adapts to various models and domains. Empirical evaluations across different classification tasks demonstrate that the suitability filter reliably detects performance deviations due to covariate shift. This enables proactive mitigation of potential failures in high-stakes applications.

Figures (8)

• Figure 1: A model $M$ is suitable for use on $D_u$ if its accuracy does not fall below the accuracy on $D_{\text{test}}$ by more than a pre-defined margin $m$. The suitability filter calculates per-sample prediction correctness probabilities for both test and user datasets and compares the two distributions through statistical non-inferiority testing. The dashed vertical lines represent the mean values of the distributions corresponding to the estimated accuracies.
• Figure 2: Schematic overview of the suitability filter. The suitability filter assesses whether model performance on a user sample $D_u$ deviates from its performance on the test dataset $D_{\text{test}}$. This is achieved by combining different suitability signals $\{s_1,\ldots,s_s\}$ to estimate per-sample prediction correctness and comparing the distribution of these estimates between the two datasets using a statistical test.
• Figure 3: Margin adjustment under accuracy estimation error. In each panel, the solid gray line is the perfect‐calibration diagonal, the dashed black/gray lines mark the original margin $m$ and its corrected value $m'$, respectively. The blue/orange arrows indicate the estimation errors on the test set ($\Delta_{\mathrm{test}}$) and user data ($\Delta_{u}$), respectively. In the left panel, the user data $D_u$ is deemed suitable; in the right panel it is deemed unsuitable.
• Figure 4: Sensitivity of suitability decisions to accuracy differences between user and test data on FMoW-WILDS. The plot, summarizing results from nearly $29k$ individual experiments, shows the percentage of SUITABLE decisions for $\alpha=0.05$ and $m=0$ across various accuracy difference bins. We combine both ID and OOD suitability filter experiments based on $3$ models trained with different random seeds.
• Figure 5: Suitability filtering on different OOD folds of FMoW-WILDS with and without additional calibration on $D_u$. We choose a non-inferiority margin of $m=0.05$ for this experiment.

Theorems & Definitions (9)

• Definition 3.1: Suitability
• Definition 3.2: Suitability Filter
• Definition 4.1: $\delta$-Calibration
• Theorem 4.2: Non-Inferiority Test Guarantee
• Lemma 4.3: Expectation of Correctness
• Corollary 4.4: Bounded False Positive Rate for Suitability Filter under $\delta$-Calibration
• Remark 4.5: Impossibility of Bounded False Positive Rate without $\delta$-Calibration
• proof
• proof