Evaluating multiple models using labeled and unlabeled data

TL;DR

SSME addresses the critical problem of evaluating multiple classifiers when large labeled datasets are unavailable by jointly modeling ground truth labels and continuous classifier scores with a semi-supervised mixture model. It leverages unlabeled data through EM, uses the additive log-ratio transform to handle bounded probability simplices, and can estimate arbitrary metrics that depend on scores and labels. Theoretical bounds show unlabeled data and multiple diverse classifiers improve performance estimation, and empirical results across healthcare, moderation, chemistry, and NLP domains demonstrate substantial reductions in metric-estimation error relative to labeling alone and to strong baselines. The approach supports case studies with large language models and subgroup analyses, highlighting practical impact for fair and robust model evaluation under label-scarce settings.

Abstract

It remains difficult to evaluate machine learning classifiers in the absence of a large, labeled dataset. While labeled data can be prohibitively expensive or impossible to obtain, unlabeled data is plentiful. Here, we introduce Semi-Supervised Model Evaluation (SSME), a method that uses both labeled and unlabeled data to evaluate machine learning classifiers. SSME is the first evaluation method to take advantage of the fact that: (i) there are frequently multiple classifiers for the same task, (ii) continuous classifier scores are often available for all classes, and (iii) unlabeled data is often far more plentiful than labeled data. The key idea is to use a semi-supervised mixture model to estimate the joint distribution of ground truth labels and classifier predictions. We can then use this model to estimate any metric that is a function of classifier scores and ground truth labels (e.g., accuracy or expected calibration error). We present experiments in four domains where obtaining large labeled datasets is often impractical: (1) healthcare, (2) content moderation, (3) molecular property prediction, and (4) image annotation. Our results demonstrate that SSME estimates performance more accurately than do competing methods, reducing error by 5.1x relative to using labeled data alone and 2.4x relative to the next best competing method. SSME also improves accuracy when evaluating performance across subsets of the test distribution (e.g., specific demographic subgroups) and when evaluating the performance of language models.

Figures (6)

• Figure 1: Using SSME with two binary classifiers. SSME applies to settings in which classifier scores are available for a set of examples, some labeled (left). SSME first fits a mixture model to estimate the joint density of scores and labels (where $s_1$ is the score assigned by the first classifier, and $s_2$ the second, middle panel). SSME next uses the mixture model to compute metrics with respect to the expected label for each example (right). SSME can be applied to any number of classifiers and classes $K$, and supports any metric that compares classifier scores to ground truth labels.
• Figure 2: Metric estimation error on binary tasks ($\mathbf{n_\ell = 20}$, $\mathbf{n_u = 1000}$). Each point plots the rescaled mean absolute error (RMAE) across runs, where 1.0 (dashed line) represents the RMAE when using labeled data alone and lower is better. SSME (gray) achieves lower estimation error than the baselines, averaging across metrics and 50 runs, reported for five binary tasks (y-axis). Results reporting mean absolute error (unscaled) and 95% CIs are in Tables \ref{['tab:binary_acc']}-\ref{['tab:binary_auprc']}.
• Figure 3: Metric estimation error for LLM classifiers ($\mathbf{n_\ell = 20}$, $\mathbf{n_u = 1000}$). SSME consistently reduces estimation error relative to labeled data alone and does so more reliably than any baseline capable of estimating performance in multiclass settings.
• Figure S1: Estimation error based on original classifier set performance. We plot estimated classifier performance as a function of original classifier performance for various values of $\epsilon_\mathbf{c}$, for accuracy (left) and AUC (right). The less accurate the classifier set, the larger the error bands.
• Figure S2: SSME performance in response to additional unlabeled data, additional classifiers, and improved classifiers. As indicated by our theoretical results, SSME benefits from increased amounts of unlabeled data, number of classifiers, and classifier performance.