What Is the Optimal Ranking Score Between Precision and Recall? We Can Always Find It and It Is Rarely $F_1$

TL;DR

The paper reframes classifier evaluation as ranking along a manifold of precision-recall tradeoffs induced by the Fβ family. It shows that Fβ-induced rankings form a geodesic between precision- (Pr) and recall-based (Re) rankings, with the optimal tradeoff found at equal Kendall distance from both ends. A closed-form expression for the optimal β is derived, based on pairwise performance comparisons, and the optimality is framed in terms of Fréchet (Karcher) means on the ranking manifold. Six case studies, including uniform distributions and real CDNet data, demonstrate that F1 often fails to yield the best ranking and provide practical adaptation rules for β, including a data-driven heuristic β^2 ≈ E[PFP]/E[PFN]. Overall, the work delivers a rigorous, principled method for choosing the ranking score between precision and recall, with broad implications for model evaluation and benchmarking.

Abstract

Ranking methods or models based on their performance is of prime importance but is tricky because performance is fundamentally multidimensional. In the case of classification, precision and recall are scores with probabilistic interpretations that are both important to consider and complementary. The rankings induced by these two scores are often in partial contradiction. In practice, therefore, it is extremely useful to establish a compromise between the two views to obtain a single, global ranking. Over the last fifty years or so,it has been proposed to take a weighted harmonic mean, known as the F-score, F-measure, or $F_β$. Generally speaking, by averaging basic scores, we obtain a score that is intermediate in terms of values. However, there is no guarantee that these scores lead to meaningful rankings and no guarantee that the rankings are good tradeoffs between these base scores. Given the ubiquity of $F_β$ scores in the literature, some clarification is in order. Concretely: (1) We establish that $F_β$-induced rankings are meaningful and define a shortest path between precision- and recall-induced rankings. (2) We frame the problem of finding a tradeoff between two scores as an optimization problem expressed with Kendall rank correlations. We show that $F_1$ and its skew-insensitive version are far from being optimal in that regard. (3) We provide theoretical tools and a closed-form expression to find the optimal value for $β$ for any distribution or set of performances, and we illustrate their use on six case studies.

Figures (69)

• Figure 1: The manifold of rankings inducible with the $\scoreFBeta$ scores is a curve (drawn as $\smile$) that depends on the set or distribution of performances that is considered. The distance along the curve is proportional to the number of swaps of neighboring classifiers needed to transform a ranking into another. The rankings induced by precision $\blacklozenge$ and recall $\blacklozenge$ are at the extremities of the manifold. The rankings induced by the traditional (balanced) $\scoreFBeta[1]$✖ and by the skew-insensitive version of $\scoreFBeta[1]$✖Flach2003TheGeometry can be anywhere, depending on the set or distribution of performances that are compared, and cannot be considered as the optimal tradeoff between precision and recall. We consider that the ranking located at equal distance, along the curve, from precision and recall is the optimal tradeoff $\blacklozenge$ to rank classifiers.
• Figure 2: Isometrics of $\scoreFBeta$ and $SIVF$ in the space $(FPR,TPR)\in[0,1]^2$, for three class priors. An isometric is a locus of equivalent performances, according to the score value. We plotted here those corresponding to the values $0.05, 0.10, 0.15, \ldots, 0.95$. The isometrics of $\scoreFBeta$ form a pencil of lines intersecting at $(FPR,TPR)=(-\ell,0)$ with $\ell=\beta^2 \pi_+/\pi_-$. The isometrics of $SIVF$ intersect at $(FPR,TPR)=(-1,0)$, no matter what the class priors are. The score $SIVF$ induces the same performance ordering as $\scoreFBeta$ with $\beta^2=\pi_-/\pi_+$.
• Figure 3: Graphical representation of some sets of two-class crisp classification performances considered in this paper. (a) All possible performances (see \ref{['eq:set-I']}) correspond to points in a regular tetrahedron. (b) The performances corresponding to fixed probabilities of true negatives (see \ref{['eq:set-II']}) are points in parallel equilateral triangles. (c) The performances corresponding to fixed class priors (see \ref{['eq:set-III']}) are points in parallel rectangles. (d) The performances corresponding to fixed class priors and "above" no-skills (see \ref{['eq:set-IV']}) are points in parallel right triangles. (e) The performances for given class priors, and "close to the oracle" (see \ref{['eq:set-V']}) are points in parallel squares. The color represents the value of the fixed parameter considered (either $ptn$ or $\pi_+$).
• Figure 4: Results for uniform distributions over the performances with fixed class priors, i.e.$\Pi_3(\pi_+)$.
• Figure 5: Results for uniform distributions over the performances with fixed class priors and close to the oracle, i.e.$\Pi_5(\pi_+)$.