Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical Multicriteria Benchmarking via the GSD-Front

Christoph Jansen, Georg Schollmeyer, Julian Rodemann, Hannah Blocher, Thomas Augustin

TL;DR

This work develops a multicriteria framework for comparing classifiers via generalized stochastic dominance, introducing the GSD-front as a more informative alternative to the Pareto-front when handling mixed ordinal and cardinal metrics. It provides a set-valued, consistent estimator for the GSD-front, and both static and dynamic permutation tests to assess membership of a classifier in the front, with robustness extensions under non-i.i.d. sampling through a $\gamma$-contamination model. The methodology is demonstrated on OpenML and PMLB benchmark suites, showing that the GSD-front yields more discriminative and stable inferences than traditional Pareto or marginal-front analyses. The approach allows practitioners to evaluate trade-offs across diverse criteria without requiring explicit weighting, while accounting for statistical uncertainty and potential data-sampling irregularities. Overall, the GSD-front offers a principled, scalable tool for reliable multicriteria benchmarking in classifier evaluation with practical implications for model selection under complex performance landscapes.

Abstract

Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.

Statistical Multicriteria Benchmarking via the GSD-Front

TL;DR

This work develops a multicriteria framework for comparing classifiers via generalized stochastic dominance, introducing the GSD-front as a more informative alternative to the Pareto-front when handling mixed ordinal and cardinal metrics. It provides a set-valued, consistent estimator for the GSD-front, and both static and dynamic permutation tests to assess membership of a classifier in the front, with robustness extensions under non-i.i.d. sampling through a -contamination model. The methodology is demonstrated on OpenML and PMLB benchmark suites, showing that the GSD-front yields more discriminative and stable inferences than traditional Pareto or marginal-front analyses. The approach allows practitioners to evaluate trade-offs across diverse criteria without requiring explicit weighting, while accounting for statistical uncertainty and potential data-sampling irregularities. Overall, the GSD-front offers a principled, scalable tool for reliable multicriteria benchmarking in classifier evaluation with practical implications for model selection under complex performance landscapes.

Abstract

Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.
Paper Structure (40 sections, 4 theorems, 33 equations, 7 figures, 1 table)

This paper contains 40 sections, 4 theorems, 33 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our contribution
  3. Related work
  4. Decision-theoretic preliminaries
  5. GSD for classifier comparison
  6. Statistical testing
  7. A test for the GSD-front
  8. Checking robustness under non-i.i.d.-scenarios
  9. Benchmarking experiments
  10. Experiments on OpenML
  11. Experiments on PMLB
  12. Concluding remarks
  13. Mathematical background
  14. Basic definitions from order theory
  15. Detailed description of the permutation test from Section \ref{['prectest']}
  16. ...and 25 more sections

Key Result

Theorem 1

Denote by $\mathcal{I}_{\Phi}$ the set of all sets $\{a: u(a) \geq c\}$, where $c \in [0,1]$ and $u \in \mathcal{U}_{\mathcal{P}_{\Phi}}$. Assume that $\succsim$ is antisymmetric. If the VC-dimensionThe VC-dimension of a set system $\mathcal{S}$ is the largest cardinality of a set $A$ with $2^A=\{A where set convergence is defined via the trivial metric.

Figures (7)

  • Figure 1: Left: Densities of resampled test statistics for pairwise permutation tests of SVM vs. six other classifiers on 80 datasets from OpenML. Big (small) vertical lines depict observed (resampled) test statistics. Rejection regions for the static (dynamic) GSD-test are highlighted red (dark red). Right: Effect of Contamination: $p$-values for pairwise tests of SVM versus GLMNet, kNN, RF and xGBoost.Red lines mark significance levels of $\alpha~=~0.05$ (dark red: $\alpha~=~\frac{0.05}{6}$). Significance of SVM being in the GSD-front remains stable under contamination of up to $7$ of $80$ datasets.
  • Figure 2: The blue shaded region symbolizes the $0$-empirical GSD-front for the OpenML data sets.
  • Figure 3: Cumulative Distribution Functions (CDFs) of resampled test statistics for hypothesis tests of SVM vs. LR, RF, kNN, GLMNet, xGBoost, and CART, respectively, on OpenML's benchmarking suite. As opposed to Figure \ref{['pairwise_tests']} in the main paper, values of observed test statistics are not included. They are: $0.0125$ (CART), $-0.3875$ (kNN), $-0.4375$ (xGBoost), $-0.41875$ (RF), $-0.3375$ (GLMNet), and $-0.04897227$ (LR). It becomes evident that the resampled test statistics' distributions for SVM vs. xGBoost and GLMNet are left-shifted compared to SVM vs. CART, xGBoost, and LR. This is also visible in Figure \ref{['pairwise_tests']} in the main paper, albeit less clearly.
  • Figure 4: Hasse graph of the empirical GSD-relation for the PMLB data sets. The blue shaded region symbolizes the $0$-empirical GSD-front, see Definition \ref{['egsd']} ii).
  • Figure 5: Densities of resampled test statistics for pairwise tests of CRE vs. six other classifiers on 62 datasets from PMLB. Big (small) vertical lines depict observed (resampled) test statistics. Rejection regions for the static (dynamic) GSD-test are highlighted red (dark red). As becomes evident, we cannot reject any of the pairwise tests for neither significance level.
  • ...and 2 more figures

Theorems & Definitions (15)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Remark 1
  • Remark 2
  • Theorem 1
  • Remark 3
  • ...and 5 more