Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Principled Algorithms for Optimizing Generalized Metrics in Binary Classification

Anqi Mao, Mehryar Mohri, Yutao Zhong

TL;DR

This work tackles optimizing generalized binary classification metrics that are ratios of linear forms in TP/FP/TN/FN under class imbalance and asymmetric costs. It reframes metric optimization as a generalized cost-sensitive learning problem, introduces a surrogate-loss family with ${\mathscr H}$-consistency guarantees, and derives finite-sample generalization bounds. The METRO algorithms offer a principled, finite-sample approach to find near-optimal predictors for these metrics via a lambda-search and convex surrogate losses, with cross-validation as a practical alternative. Experiments on CIFAR-10/100 and SVHN demonstrate consistent gains over prior baselines, validating the framework's theoretical and practical benefits for metric-aware binary classification.

Abstract

In applications with significant class imbalance or asymmetric costs, metrics such as the $F_β$-measure, AM measure, Jaccard similarity coefficient, and weighted accuracy offer more suitable evaluation criteria than standard binary classification loss. However, optimizing these metrics present significant computational and statistical challenges. Existing approaches often rely on the characterization of the Bayes-optimal classifier, and use threshold-based methods that first estimate class probabilities and then seek an optimal threshold. This leads to algorithms that are not tailored to restricted hypothesis sets and lack finite-sample performance guarantees. In this work, we introduce principled algorithms for optimizing generalized metrics, supported by $H$-consistency and finite-sample generalization bounds. Our approach reformulates metric optimization as a generalized cost-sensitive learning problem, enabling the design of novel surrogate loss functions with provable $H$-consistency guarantees. Leveraging this framework, we develop new algorithms, METRO (Metric Optimization), with strong theoretical performance guarantees. We report the results of experiments demonstrating the effectiveness of our methods compared to prior baselines.

Principled Algorithms for Optimizing Generalized Metrics in Binary Classification

TL;DR

This work tackles optimizing generalized binary classification metrics that are ratios of linear forms in TP/FP/TN/FN under class imbalance and asymmetric costs. It reframes metric optimization as a generalized cost-sensitive learning problem, introduces a surrogate-loss family with -consistency guarantees, and derives finite-sample generalization bounds. The METRO algorithms offer a principled, finite-sample approach to find near-optimal predictors for these metrics via a lambda-search and convex surrogate losses, with cross-validation as a practical alternative. Experiments on CIFAR-10/100 and SVHN demonstrate consistent gains over prior baselines, validating the framework's theoretical and practical benefits for metric-aware binary classification.

Abstract

In applications with significant class imbalance or asymmetric costs, metrics such as the -measure, AM measure, Jaccard similarity coefficient, and weighted accuracy offer more suitable evaluation criteria than standard binary classification loss. However, optimizing these metrics present significant computational and statistical challenges. Existing approaches often rely on the characterization of the Bayes-optimal classifier, and use threshold-based methods that first estimate class probabilities and then seek an optimal threshold. This leads to algorithms that are not tailored to restricted hypothesis sets and lack finite-sample performance guarantees. In this work, we introduce principled algorithms for optimizing generalized metrics, supported by -consistency and finite-sample generalization bounds. Our approach reformulates metric optimization as a generalized cost-sensitive learning problem, enabling the design of novel surrogate loss functions with provable -consistency guarantees. Leveraging this framework, we develop new algorithms, METRO (Metric Optimization), with strong theoretical performance guarantees. We report the results of experiments demonstrating the effectiveness of our methods compared to prior baselines.
Paper Structure (21 sections, 22 theorems, 74 equations, 1 figure, 2 tables, 3 algorithms)

This paper contains 21 sections, 22 theorems, 74 equations, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Formulation
  4. Generalized Metrics
  5. Equivalent Problem
  6. General Cost-Sensitive Learning
  7. General Target Loss
  8. General Surrogate Losses
  9. Theoretical Guarantees
  10. Algorithm for Generalized Metrics
  11. Experiments
  12. Conclusion
  13. Related work
  14. Proof of Theorem \ref{['thm:equiv-non']}
  15. Proof of Theorem \ref{['thm:bound']}
  16. ...and 6 more sections

Key Result

Theorem 3.1

The equality $\mathcal{L}(h^*) = \mathcal{L}^*({\mathscr H})$ holds for $h^* \in {\mathscr H}$ if and only if ${\mathscr E}_{\ell^{\lambda^*\!\!}}(h^*) = {\mathscr E}_{\ell^{\lambda^*\!\!}}({\mathscr H}) = 0$.

Figures (1)

  • Figure 1: Comparison of the best-in-class SVM classifier, the optimal linear hypothesis for the $F_{0.5}$-measure, and the linear hypothesis returned by our algorithm.

Theorems & Definitions (36)

  • Definition 2.1: Bayes-Consistency
  • Definition 2.2: ${\mathscr H}$-Consistency bound
  • Theorem 3.1
  • proof
  • Theorem 3.2
  • Theorem 4.1
  • Corollary 4.2
  • Corollary 4.3
  • Theorem 5.1
  • Theorem 5.2
  • ...and 26 more