Enhancing Classifier Evaluation: A Fairer Benchmarking Strategy Based on Ability and Robustness

TL;DR

The paper tackles the problem of fair classifier evaluation by jointly modeling dataset difficulty and classifier ability using Item Response Theory (IRT) and a Glicko-2 tournament-style rating system. It introduces decodIRT, a workflow that fits a 3PL IRT model to ML data, computes True-Score, and then updates classifier ratings via round-robin matches across datasets. A case study on OpenML-CC18 reveals that only a minority of datasets are truly difficult, and a subset (about 50%) can retain similar evaluation power while reducing benchmark size; Random Forest often shows the strongest ability. The proposed IRT-Glicko framework provides a nuanced, robust method for benchmarking that accounts for both data complexity and model proficiency, with practical implications for designing more informative benchmarks and understanding dataset-model interactions.

Abstract

Benchmarking is a fundamental practice in machine learning (ML) for comparing the performance of classification algorithms. However, traditional evaluation methods often overlook a critical aspect: the joint consideration of dataset complexity and an algorithm's ability to generalize. Without this dual perspective, assessments may favor models that perform well on easy instances while failing to capture their true robustness. To address this limitation, this study introduces a novel evaluation methodology that combines Item Response Theory (IRT) with the Glicko-2 rating system, originally developed to measure player strength in competitive games. IRT assesses classifier ability based on performance over difficult instances, while Glicko-2 updates performance metrics - such as rating, deviation, and volatility - via simulated tournaments between classifiers. This combined approach provides a fairer and more nuanced measure of algorithm capability. A case study using the OpenML-CC18 benchmark showed that only 15% of the datasets are truly challenging and that a reduced subset with 50% of the original datasets offers comparable evaluation power. Among the algorithms tested, Random Forest achieved the highest ability score. The results highlight the importance of improving benchmark design by focusing on dataset quality and adopting evaluation strategies that reflect both difficulty and classifier proficiency.

Figures (16)

• Figure 1: Example of Item Characteristic Curve.
• Figure 2: Flowchart of the proposed methodology.
• Figure 3: Flowchart of the first step.
• Figure 4: Flowchart of the second step.
• Figure 5: Flowchart of the third step.