Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Macro F1 and Macro F1

Juri Opitz, Sebastian Burst

TL;DR

Two macro F1 definitions are compared: Averaged F1 (the arithmetic mean of per-class F1 scores) and F1 of averages (the harmonic mean of the average precision and recall). The authors prove a closed-form expression for the difference $Delta$ and show $Delta$ can reach $0.5$ when the number of classes $n$ is even, or $0.5 - 1/(2 n^2)$ otherwise. They also provide a Theorem and a Lemma establishing when equality holds and how class bias affects the gap. Through numerical experiments, they show that the metrics can yield different values and even reverse rankings, especially on imbalanced data, and conclude that Averaged F1 is more robust and advise reporting the chosen formula or preferring Averaged F1 for macro evaluation.

Abstract

The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalent. More specifically, one formula well 'rewards' classifiers which produce a skewed error type distribution. In fact, the difference in outcome of the two computations can be as high as 0.5. The two computations may not only diverge in their scalar result but can also lead to different classifier rankings.

Macro F1 and Macro F1

TL;DR

Two macro F1 definitions are compared: Averaged F1 (the arithmetic mean of per-class F1 scores) and F1 of averages (the harmonic mean of the average precision and recall). The authors prove a closed-form expression for the difference and show can reach when the number of classes is even, or otherwise. They also provide a Theorem and a Lemma establishing when equality holds and how class bias affects the gap. Through numerical experiments, they show that the metrics can yield different values and even reverse rankings, especially on imbalanced data, and conclude that Averaged F1 is more robust and advise reporting the chosen formula or preferring Averaged F1 for macro evaluation.

Abstract

The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalent. More specifically, one formula well 'rewards' classifiers which produce a skewed error type distribution. In fact, the difference in outcome of the two computations can be as high as 0.5. The two computations may not only diverge in their scalar result but can also lead to different classifier rankings.

Paper Structure

This paper contains 16 sections, 15 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Averaged F1: arithmetic mean over harmonic means
  4. F1 of averages: harmonic mean over arithmetic means
  5. Mathematical analysis
  6. Theorem
  7. Lemma
  8. Numerical experiments
  9. Bias in data
  10. Balanced data sets
  11. Discussion and conclusion
  12. Proof Lemma
  13. Proof Theorem 2.
  14. Proof Theorem 3.
  15. Implementation example
  16. ...and 1 more sections

Figures (3)

  • Figure 1: (a): Macro F1 results from 1000 randomly sampled binary classification tasks with class distribution 95% vs. 5% and random classifier. (b): $\Delta$ density (KDE) from 1000 randomly sampled classification tasks with respect to various class distributions (see legend) and random classifier.
  • Figure 2: $\Delta$ for random classification tasks on 2000 data points with different classifier performance ($x$-axis) and different label distribution ($y$-axis).
  • Figure 3: $\Delta$ for random classification tasks on 2000 data points with different classifier performance ($x$-axis) and classification probability for non-gold labels ($y$-axis).