Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Who is the Winning Algorithm? Rank Aggregation for Comparative Studies

Amichai Painsky

TL;DR

The paper tackles identifying the best-performing algorithm by estimating win probabilities from full ranking data across datasets. It generalizes the conventional MLE (PAMA) by introducing a weighted rank-aggregation estimator $\hat{p}^w$, with both data-independent minimax and data-dependent leave-one-out schemes. Theoretical results provide worst-case $D_{TV}$ bounds and extend to top-$K$ ranks, while empirical results on synthetic and real benchmarks (TabZilla, Fernández- Delgado, Shmuel) show the data-dependent approach improves accuracy and robustness over MLE. The work offers practical guidance for comparative studies and broadens rank-aggregation methods to real-world algorithm selection tasks.

Abstract

Consider a collection of m competing machine learning algorithms. Given their performance on a benchmark of datasets, we would like to identify the best performing algorithm. Specifically, which algorithm is most likely to ``win'' (rank highest) on a future, unseen dataset. The standard maximum likelihood approach suggests counting the number of wins per each algorithm. In this work, we argue that there is much more information in the complete rankings. That is, the number of times that each algorithm finished second, third and so forth. Yet, it is not entirely clear how to effectively utilize this information for our purpose. In this work we introduce a novel conceptual framework for estimating the win probability for each of the m algorithms, given their complete rankings over a benchmark of datasets. Our proposed framework significantly improves upon currently known methods in synthetic and real-world examples.

Who is the Winning Algorithm? Rank Aggregation for Comparative Studies

TL;DR

The paper tackles identifying the best-performing algorithm by estimating win probabilities from full ranking data across datasets. It generalizes the conventional MLE (PAMA) by introducing a weighted rank-aggregation estimator , with both data-independent minimax and data-dependent leave-one-out schemes. Theoretical results provide worst-case bounds and extend to top- ranks, while empirical results on synthetic and real benchmarks (TabZilla, Fernández- Delgado, Shmuel) show the data-dependent approach improves accuracy and robustness over MLE. The work offers practical guidance for comparative studies and broadens rank-aggregation methods to real-world algorithm selection tasks.

Abstract

Consider a collection of m competing machine learning algorithms. Given their performance on a benchmark of datasets, we would like to identify the best performing algorithm. Specifically, which algorithm is most likely to ``win'' (rank highest) on a future, unseen dataset. The standard maximum likelihood approach suggests counting the number of wins per each algorithm. In this work, we argue that there is much more information in the complete rankings. That is, the number of times that each algorithm finished second, third and so forth. Yet, it is not entirely clear how to effectively utilize this information for our purpose. In this work we introduce a novel conceptual framework for estimating the win probability for each of the m algorithms, given their complete rankings over a benchmark of datasets. Our proposed framework significantly improves upon currently known methods in synthetic and real-world examples.
Paper Structure (12 sections, 4 theorems, 46 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 12 sections, 4 theorems, 46 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Definitions and Problem Statement
  4. The Data Independent Scheme
  5. The Data Dependent Scheme
  6. Experiments
  7. Synthetic Experiments
  8. Real-world Experiments
  9. Proofs
  10. A proof for Theorem \ref{['T1']}
  11. A proof for Theorem \ref{['T3']}
  12. Conclusions

Key Result

Theorem 1

Let $p \in \Delta_m$ be the win probability for each of the algorithms. Let $\hat{p}^w=(wr^{(1)}+(1-w)r^{(2)})/n$ be our proposed minimax estimator. Then, Further, the weight $w^*$ which minimizes (ub_T1) is given by

Figures (4)

  • Figure 1: Data-independent upper bounds for $m=5$, under the expected TV distance
  • Figure 2: Comparing Algorithm $1$ (blue) with the MLE (red) and an Oracle estimator (black) in two synthetic experiments. We use the following parameters: Zipf's Law: $s=1.01$, Geometric: $\alpha=0.4$, Negative-Binomial: $l=1, r=0.003$, Beta-Binomial: $\alpha=\beta=2$.
  • Figure 3: The obtained weights of Algorithm $1$ (upper charts) and the Oracle (lower charts) in the synthetic experiment above (Figure \ref{['fig1']})
  • Figure 4: Comparing the data-independent schemes (Theorems \ref{['T1']} and \ref{['T3']}) with the MLE (red) and an Oracle estimator (black) in small sample synthetic experiments. The distributions' parameters are described in Figure \ref{['fig1']}.

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2
  • Proposition 1
  • proof
  • Proposition 2
  • proof