Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking

TL;DR

A unified and practical framework for unsupervised model evaluation and ranking in two common deployment settings: estimating the accuracy of a fixed model on multiple unlabeled test sets (dataset-centric evaluation), and ranking a set of candidate models on a single unlabeled test set (model-centric evaluation).

Abstract

Assessing model generalization under distribution shift is essential for real-world deployment, particularly when labeled test data is unavailable. This paper presents a unified and practical framework for unsupervised model evaluation and ranking in two common deployment settings: (1) estimating the accuracy of a fixed model on multiple unlabeled test sets (dataset-centric evaluation), and (2) ranking a set of candidate models on a single unlabeled test set (model-centric evaluation). We demonstrate that two intrinsic properties of model predictions, namely confidence (which reflects prediction certainty) and dispersity (which captures the diversity of predicted classes), together provide strong and complementary signals for generalization. We systematically benchmark a set of confidence-based, dispersity-based, and hybrid metrics across a wide range of model architectures, datasets, and distribution shift types. Our results show that hybrid metrics consistently outperform single-aspect metrics on both dataset-centric and model-centric evaluation settings. In particular, the nuclear norm of the prediction matrix provides robust and accurate performance across tasks, including real-world datasets, and maintains reliability under moderate class imbalance. These findings offer a practical and generalizable basis for unsupervised model assessment in deployment scenarios.

Figures (11)

• Figure 1: Illustration of the two unsupervised generalization analysis tasks. (a) Dataset-Centric Evaluation: A fixed model is evaluated on a collection of unlabeled test sets drawn from diverse distributions. The objective is to estimate its generalization performance on each distribution without access to labeled data. (b) Model-Centric Evaluation: A single unlabeled test set is used to compare multiple candidate models. The goal is to rank the models by their expected performance on the target distribution without relying on test-time supervision. These two complementary setups enable a comprehensive understanding of model generalization under distribution shift in a label-free manner.
• Figure 2: Average Spearman's Rank Correlation coefficient $\rho$ of each metric across ImageNet-C, ImageNet-3D, CIFAR-10, and WILDS setups in dataset-centric evaluation. Each bar shows the correlation ($\rho$) between a metric and model accuracy across multiple test sets. Metrics are grouped into three categories: (i) Confidence-based (e.g., ATC, DoC), which may perform well on clean data but degrade under distribution shift; (ii) Dispersity-based (e.g., ClassEntropy, CTD), which capture prediction variation across classes and are more robust across datasets; (iii) Hybrid metrics (e.g., NuclearNorm, COT), which combine confidence and dispersity and consistently achieve the strongest alignment with true accuracy rankings. Note: CTD and COT are negatively correlated with accuracy by design, so $-\rho$ is shown for comparability.
• Figure 3: Correlation study under the ImageNet setup. We plot the actual accuracy of DenseNet against predictions from 12 methods. Each shape in a subfigure denotes a test set, and the solid lines represent linear fits on synthetic ImageNet-C subsets. We mark six real-world datasets with arrows and summarize the 19 ImageNet-C corruption types at the top using distinct shape–color pairs. While metrics (e.g., ConfScore and ClassEntropy) exhibit noisy trends, COT and especially NuclearNorm show strong, consistent alignment with accuracy, with NuclearNorm yielding the closest fit to the regression line.
• Figure 4: Correlation study under the CIFAR-10 setup. We plot the actual accuracy of ResNet-20 and the estimated OOD quantity. We show the results of nuclear norm, ConfScore, ClassEntropy, and COT. The lines are calculated by the linear regression fit on CIFAR-C. We mark the $3$ real-world test sets in each sub-figure.
• Figure 5: Correlation study under the CUB-200 setup. We plot the actual accuracy of ResNet-50 and the estimated OOD quantity. We show results of nuclear norm, ConfScore, ClassEntropy, and COT The straight lines are calculated by the linear regression fit on CUB-200-C. We mark the real-world test set CUB-P in each sub-figure.