On Evaluation of Vision Datasets and Models using Human Competency Frameworks

TL;DR

This paper explores Item Response Theory (IRT), a framework that infers interpretable latent parameters for an ensemble of models and each dataset item, enabling richer evaluation and analysis beyond the single accuracy number.

Abstract

Evaluating models and datasets in computer vision remains a challenging task, with most leaderboards relying solely on accuracy. While accuracy is a popular metric for model evaluation, it provides only a coarse assessment by considering a single model's score on all dataset items. This paper explores Item Response Theory (IRT), a framework that infers interpretable latent parameters for an ensemble of models and each dataset item, enabling richer evaluation and analysis beyond the single accuracy number. Leveraging IRT, we assess model calibration, select informative data subsets, and demonstrate the usefulness of its latent parameters for analyzing and comparing models and datasets in computer vision.

Figures (10)

• Figure 1: 3PL ICC for image with $b=5$
• Figure 2: Overall Workflow
• Figure 3: Percentage of images with annotation errors and class overlap for given values of overconfidence across different models: (Top) ResNet-18 (20 epochs) (Middle) ResNet-50 (100 epochs) (Bottom) ViT
• Figure 4: Class-wise median guessing vs. class-wise median difficulty and discriminability of the Gaussian noise corruption: (Top) Severity 1 (Bottom) Severity 5
• Figure 5: Correlation of rankings on small subset with overall rankings