Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online AUC Optimization Based on Second-order Surrogate Loss

JunRu Luo, Difei Cheng, Bo Zhang

TL;DR

This work targets online AUC maximization for imbalanced binary classification by introducing a second-order surrogate loss derived from first- and second-order data moments. It replaces the aggregated pairwise AUC loss with a moment-based surrogate, enabling memory-efficient online updates and sublinear regret. The authors develop OAUC-SOSL for linear models with $O(p^2)$ space and prove an $O(\ln T)$ regret bound for the hinge-based variant, plus kernelized extensions OKAUC-M with analogous guarantees. Extensive experiments across multiple datasets show competitive AUC performance and improved efficiency, especially with kernelized implementations. The framework offers a principled bridge between hinge-like margin optimization and moment-based statistics, with potential extensions to distributionally robust and nonlinear learning settings.

Abstract

The Area Under the Curve (AUC) is an important performance metric for classification tasks, particularly in class-imbalanced scenarios. However, minimizing the AUC presents significant challenges due to the non-convex and discontinuous nature of pairwise 0/1 losses, which are difficult to optimize, as well as the substantial memory cost of instance-wise storage, which creates bottlenecks in large-scale applications. To overcome these challenges, we propose a novel second-order surrogate loss based on the pairwise hinge loss, and develop an efficient online algorithm. Unlike conventional approaches that approximate each individual pairwise 0/1 loss term with an instance-wise surrogate function, our approach introduces a new paradigm that directly substitutes the entire aggregated pairwise loss with a surrogate loss function constructed from the first- and second-order statistics of the training data. Theoretically, while existing online AUC optimization algorithms typically achieve an $\mathcal{O}(\sqrt{T})$ regret bound, our method attains a tighter $\mathcal{O}(\ln T)$ bound. Furthermore, we extend the proposed framework to nonlinear settings through a kernel-based formulation. Extensive experiments on multiple benchmark datasets demonstrate the superior efficiency and effectiveness of the proposed second-order surrogate loss in optimizing online AUC performance.

Online AUC Optimization Based on Second-order Surrogate Loss

TL;DR

This work targets online AUC maximization for imbalanced binary classification by introducing a second-order surrogate loss derived from first- and second-order data moments. It replaces the aggregated pairwise AUC loss with a moment-based surrogate, enabling memory-efficient online updates and sublinear regret. The authors develop OAUC-SOSL for linear models with space and prove an regret bound for the hinge-based variant, plus kernelized extensions OKAUC-M with analogous guarantees. Extensive experiments across multiple datasets show competitive AUC performance and improved efficiency, especially with kernelized implementations. The framework offers a principled bridge between hinge-like margin optimization and moment-based statistics, with potential extensions to distributionally robust and nonlinear learning settings.

Abstract

The Area Under the Curve (AUC) is an important performance metric for classification tasks, particularly in class-imbalanced scenarios. However, minimizing the AUC presents significant challenges due to the non-convex and discontinuous nature of pairwise 0/1 losses, which are difficult to optimize, as well as the substantial memory cost of instance-wise storage, which creates bottlenecks in large-scale applications. To overcome these challenges, we propose a novel second-order surrogate loss based on the pairwise hinge loss, and develop an efficient online algorithm. Unlike conventional approaches that approximate each individual pairwise 0/1 loss term with an instance-wise surrogate function, our approach introduces a new paradigm that directly substitutes the entire aggregated pairwise loss with a surrogate loss function constructed from the first- and second-order statistics of the training data. Theoretically, while existing online AUC optimization algorithms typically achieve an regret bound, our method attains a tighter bound. Furthermore, we extend the proposed framework to nonlinear settings through a kernel-based formulation. Extensive experiments on multiple benchmark datasets demonstrate the superior efficiency and effectiveness of the proposed second-order surrogate loss in optimizing online AUC performance.
Paper Structure (21 sections, 8 theorems, 100 equations, 2 tables, 4 algorithms)

This paper contains 21 sections, 8 theorems, 100 equations, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Online AUC Optimization Problems
  3. Second-Order Surrogate Loss Function
  4. Second-Order Surrogate Loss for Square-Based AUC Optimization
  5. Second-Order Surrogate Loss for Hinge-Based AUC Optimization
  6. Properties of the Second-Order Surrogate Loss
  7. Online AUC Maximization Based on Second-Order Surrogate Loss
  8. Online Gradient Descent Algorithm
  9. Online Update of First and Second-Order Statistics
  10. Regret Analysis of OAUC-SOSL Algorithms
  11. Online nonlinear AUC maximization
  12. online kernel AUC maximization method
  13. Update Classifier
  14. Update Buffer
  15. Regret Analysis
  16. ...and 6 more sections

Key Result

Theorem 1

Let $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^n \subset \mathbb{R}^p \times \{-1, +1\}$ be a set of training instances with labels opposite to a given example $(\mathbf{x}, y)$, and let $\mathbf{w} \in \mathbb{R}^p \setminus {\mathbf{0}}$ be an arbitrary weight vector. Denote the empirical mean a Then, the hinge-based AUC loss admits the following upper bound:

Theorems & Definitions (8)

  • Theorem 1: Moment-Constrained Upper Bound for Hinge-based AUC Loss
  • Lemma 2: Constrained Hinge Loss Bound
  • Lemma 3: Approximation Properties of $\psi_M$
  • Lemma 4: Properties of $\Psi_M$
  • Theorem 5: Regret Bound for OAUC-S
  • Theorem 6: Regret Bound for OAUC-M
  • Lemma 7
  • Theorem 8