Ranking with Ties based on Noisy Performance Data

TL;DR

This paper tackles ranking objects when performance measurements are noisy, causing overlapping intervals and incomparability, by introducing partial rankings based on a strict partial order $<_{\mathbf{P}}$. It develops three core methodologies: (1) an DAG-depth approach to obtain an arbitrary number of ranks, (2) a reordering strategy to reduce the number of ranks, and (3) a minimum-rank construction via incomparability graphs, along with implications for special cases. It further extends the framework to quantile-based comparisons, introducing reliability measures across multiple quantile limits to select the most robust ranking. The authors demonstrate the approach on Generalized Least Squares (GLS) algorithm variants and a real-world Purchase-to-Pay process dataset, showing how partial rankings facilitate root-cause analysis and provide insights beyond traditional median-based comparisons. Overall, the work offers a formal, data-driven toolkit for extracting meaningful performance class structures from noisy data and supports practical decision-making in HPC and business-process contexts.

Abstract

We consider the problem of ranking a set of objects based on their performance when the measurement of said performance is subject to noise. In this scenario, the performance is measured repeatedly, resulting in a range of measurements for each object. If the ranges of two objects do not overlap, then we consider one object as 'better' than the other, and we expect it to receive a higher rank; if, however, the ranges overlap, then the objects are incomparable, and we wish them to be assigned the same rank. Unfortunately, the incomparability relation of ranges is in general not transitive; as a consequence, in general the two requirements cannot be satisfied simultaneously, i.e., it is not possible to guarantee both distinct ranks for objects with separated ranges, and same rank for objects with overlapping ranges. This conflict leads to more than one reasonable way to rank a set of objects. In this paper, we explore the ambiguities that arise when ranking with ties, and define a set of reasonable rankings, which we call partial rankings. We develop and analyse three different methodologies to compute a partial ranking. Finally, we show how performance differences among objects can be investigated with the help of partial ranking.

Figures (22)

• Figure 1: The execution time measurements of ten algorithmic variants to solve the GLS problem: $(X^{T}M^{-1}X)^{-1}X^{T}M^{-1}\mathbf{y}$ where $X \in \mathbb{R}^{1000 \times 100}$, $M \in \mathbb{R}^{1000 \times 1000}$ and $\mathbf{y} \in \mathbb{R}^{1000}$. For each variant, the execution times are shown as box plots; red lines indicate the median values; the box indicates the Inter-Quartile Interval.
• Figure 2: Sets of measurements ($\mathcal{M}_0$).
• Figure 3: When distributions overlap, the partial ordering of the algorithms admits many reasonable rankings.
• Figure 4: When $<_{\mathbf{P}}$ imposes either a linear order or a weak order on $\mathcal{M}$, then we want the ranking to be unique. Moreover, when the order is linear, then we want the ranking to have no ties.
• Figure 5: Undirected graph of the objects in $\mathcal{M}_2$ associated by $\sim$ according to $<_{eg}$.

Theorems & Definitions (8)

• Definition 1
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof
• Definition 2