Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A General Online Algorithm for Optimizing Complex Performance Metrics

Wojciech Kotłowski, Marek Wydmuch, Erik Schultheis, Rohit Babbar, Krzysztof Dembczyński

TL;DR

This paper introduces and analyzes a general online algorithm that can be used in a straightforward way with a variety of complex performance metrics in binary, multi-class, and multi-label classification problems and shows the algorithm attains regret for concave and smooth metrics.

Abstract

We consider sequential maximization of performance metrics that are general functions of a confusion matrix of a classifier (such as precision, F-measure, or G-mean). Such metrics are, in general, non-decomposable over individual instances, making their optimization very challenging. While they have been extensively studied under different frameworks in the batch setting, their analysis in the online learning regime is very limited, with only a few distinguished exceptions. In this paper, we introduce and analyze a general online algorithm that can be used in a straightforward way with a variety of complex performance metrics in binary, multi-class, and multi-label classification problems. The algorithm's update and prediction rules are appealingly simple and computationally efficient without the need to store any past data. We show the algorithm attains $\mathcal{O}(\frac{\ln n}{n})$ regret for concave and smooth metrics and verify the efficiency of the proposed algorithm in empirical studies.

A General Online Algorithm for Optimizing Complex Performance Metrics

TL;DR

This paper introduces and analyzes a general online algorithm that can be used in a straightforward way with a variety of complex performance metrics in binary, multi-class, and multi-label classification problems and shows the algorithm attains regret for concave and smooth metrics.

Abstract

We consider sequential maximization of performance metrics that are general functions of a confusion matrix of a classifier (such as precision, F-measure, or G-mean). Such metrics are, in general, non-decomposable over individual instances, making their optimization very challenging. While they have been extensively studied under different frameworks in the batch setting, their analysis in the online learning regime is very limited, with only a few distinguished exceptions. In this paper, we introduce and analyze a general online algorithm that can be used in a straightforward way with a variety of complex performance metrics in binary, multi-class, and multi-label classification problems. The algorithm's update and prediction rules are appealingly simple and computationally efficient without the need to store any past data. We show the algorithm attains regret for concave and smooth metrics and verify the efficiency of the proposed algorithm in empirical studies.
Paper Structure (24 sections, 3 theorems, 100 equations, 6 figures, 6 tables, 2 algorithms)

This paper contains 24 sections, 3 theorems, 100 equations, 6 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work.
  3. Problem Setup
  4. Regret.
  5. The algorithm
  6. Motivation.
  7. Example: binary classification.
  8. Example: multi-label classification (with a budget).
  9. Theoretical analysis
  10. Alternative variants of the algorithm
  11. Internal semi-empirical confusion matrix.
  12. Sparse variant for a large number of labels.
  13. Empirical study
  14. Discussion
  15. Motivation for the OMMA algorithm
  16. ...and 9 more sections

Key Result

Theorem 4.3

Let as:utility hold. Then the OMMA algorithm has its regret bounded by: where $a=1, b=1$ for multi-class, and $a=m, b=\sqrt{2}$ for multi-label classification.

Figures (6)

  • Figure 1: The online protocol.
  • Figure 2: Comparison of the incremental performance of the online algorithms on the Flickr dataset. Averaged over 5 runs, the opaque fill indicates the standard deviation at given iteration $t$.
  • Figure 3: Impact of $\lambda$ on the results of the online algorithms on the Flickr dataset. Averaged over 5 runs.
  • Figure 4: Running comparison of performance for the online algorithms. Averaged over 5 runs, the opaque fill indicate the standard deviation at given iteration $t$.
  • Figure 5: Comparison of performance of the online algorithms with different values of $\lambda$. Averaged over 5 runs.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Remark 2.1: Adversarial sequence of inputs
  • Remark 4.2
  • Theorem 4.3
  • Remark 4.4
  • Theorem 5.1
  • proof
  • Lemma 5.2
  • proof