Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foundations of the Theory of Performance-Based Ranking

Sébastien Piérard, Anaïs Halin, Anthony Cioppa, Adrien Deliège, Marc Van Droogenbroeck

TL;DR

The paper addresses the lack of theoretical grounding in performance-based ranking by introducing a universal six-element mathematical framework that fuses probability and order theory. It defines performance as a probability measure, models tasks via a satisfaction variable, and captures evaluation and application-specific preferences through a coherent system of scores and the ranking scores rankingScore. The authors present three axioms and three sufficient-condition theorems, establishing that the ranking scores induce orderings satisfying these axioms and are applicable to any task, including two-class classification where they recover classic metrics like accuracy, recall, and precision while highlighting unsuitable alternatives. This framework provides a principled basis for sound, preference-aware rankings, with practical implications for designing challenges and selecting appropriate metrics. The work thereby advances a universal, theory-grounded approach to ranking classifiers, detectors, and related entities, supporting diverse applications and avoiding misleading or unstable rankings.

Abstract

Ranking entities such as algorithms, devices, methods, or models based on their performances, while accounting for application-specific preferences, is a challenge. To address this challenge, we establish the foundations of a universal theory for performance-based ranking. First, we introduce a rigorous framework built on top of both the probability and order theories. Our new framework encompasses the elements necessary to (1) manipulate performances as mathematical objects, (2) express which performances are worse than or equivalent to others, (3) model tasks through a variable called satisfaction, (4) consider properties of the evaluation, (5) define scores, and (6) specify application-specific preferences through a variable called importance. On top of this framework, we propose the first axiomatic definition of performance orderings and performance-based rankings. Then, we introduce a universal parametric family of scores, called ranking scores, that can be used to establish rankings satisfying our axioms, while considering application-specific preferences. Finally, we show, in the case of two-class classification, that the family of ranking scores encompasses well-known performance scores, including the accuracy, the true positive rate (recall, sensitivity), the true negative rate (specificity), the positive predictive value (precision), and F1. However, we also show that some other scores commonly used to compare classifiers are unsuitable to derive performance orderings satisfying the axioms.

Foundations of the Theory of Performance-Based Ranking

TL;DR

The paper addresses the lack of theoretical grounding in performance-based ranking by introducing a universal six-element mathematical framework that fuses probability and order theory. It defines performance as a probability measure, models tasks via a satisfaction variable, and captures evaluation and application-specific preferences through a coherent system of scores and the ranking scores rankingScore. The authors present three axioms and three sufficient-condition theorems, establishing that the ranking scores induce orderings satisfying these axioms and are applicable to any task, including two-class classification where they recover classic metrics like accuracy, recall, and precision while highlighting unsuitable alternatives. This framework provides a principled basis for sound, preference-aware rankings, with practical implications for designing challenges and selecting appropriate metrics. The work thereby advances a universal, theory-grounded approach to ranking classifiers, detectors, and related entities, supporting diverse applications and avoiding misleading or unstable rankings.

Abstract

Ranking entities such as algorithms, devices, methods, or models based on their performances, while accounting for application-specific preferences, is a challenge. To address this challenge, we establish the foundations of a universal theory for performance-based ranking. First, we introduce a rigorous framework built on top of both the probability and order theories. Our new framework encompasses the elements necessary to (1) manipulate performances as mathematical objects, (2) express which performances are worse than or equivalent to others, (3) model tasks through a variable called satisfaction, (4) consider properties of the evaluation, (5) define scores, and (6) specify application-specific preferences through a variable called importance. On top of this framework, we propose the first axiomatic definition of performance orderings and performance-based rankings. Then, we introduce a universal parametric family of scores, called ranking scores, that can be used to establish rankings satisfying our axioms, while considering application-specific preferences. Finally, we show, in the case of two-class classification, that the family of ranking scores encompasses well-known performance scores, including the accuracy, the true positive rate (recall, sensitivity), the true negative rate (specificity), the positive predictive value (precision), and F1. However, we also show that some other scores commonly used to compare classifiers are unsuitable to derive performance orderings satisfying the axioms.

Paper Structure

This paper contains 70 sections, 28 theorems, 49 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mathematical Framework
  4. The Performance $P$ as a Probability Measure
  5. Ordering Performances with a Preorder $\lesssim$
  6. Modeling the Task as a Random Variable $S$
  7. Modeling the Evaluation as a Function $\Phi$
  8. The Scores as Functions $X$ of Performances
  9. Modeling Application-Specific Preferences as a Random Variable $I$
  10. Performance-Based Ranking Theory
  11. Axiomatic Definition
  12. Leveraging the Preorder $\lesssim$
  13. Leveraging the Satisfaction $S$
  14. Leveraging the Combinations $\Phi$
  15. Consistency
  16. ...and 55 more sections

Key Result

Corollary 1

For any $s$, all performances $P$ such that $P(S=s)=1$ are either equivalent or incomparable.

Figures (2)

  • Figure 1: This work establishes the foundations of the theory of performance-based ranking. We do this in two steps. First, we introduce a new mathematical framework with $6$ main elements, as depicted here by the pillars. Second, we build on top of it: (1) a set of three axioms for the ordering of performances and for the performance-based ranking of entities, (2) sufficient conditions for them when the performance ordering is induced by a score, and (3) a family of scores, named ranking scores that consider the application-specific preferences. This theory is universal in the sense that it is applicable to any task.
  • Figure A.7.1: Passages between two formulations (left: classical, right: ours) for the performance analysis of two-class classification problems.

Theorems & Definitions (59)

  • Example 1
  • Example 1: continued
  • Example 1: continued
  • Example 1: continued
  • Example 1: continued
  • Example 1: continued
  • Example 1: continued
  • Corollary 1
  • Corollary 2
  • Corollary 3
  • ...and 49 more