Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation

Wentao Shi, Chenxu Wang, Fuli Feng, Yang Zhang, Wenjie Wang, Junkang Wu, Xiangnan He

TL;DR

This work tackles the gap between efficient optimization metrics and Top-K–oriented evaluation in recommender systems. It introduces Lower-Left Partial AUC (LLPAUC), a constrained partial AUC that focuses on the Lower-Left ROC region to prioritize top-ranked items while retaining AUC-like efficiency. The authors provide theoretical results showing LLPAUC's tighter correlation with Recall@K and Precision@K than AUC or OPAUC, and they develop a differentiable, minimax, point-wise loss using an Average Top-K trick to maximize LLPAUC efficiently. Extensive experiments on multiple real-world datasets demonstrate LLPAUC’s superior performance and robustness to noisy feedback across different backbones, with convergence comparable to existing losses. The work offers a practical, scalable optimization objective that aligns closely with Top-K goals, enabling better large-scale recommendations in real-world systems.

Abstract

Optimization metrics are crucial for building recommendation systems at scale. However, an effective and efficient metric for practical use remains elusive. While Top-K ranking metrics are the gold standard for optimization, they suffer from significant computational overhead. Alternatively, the more efficient accuracy and AUC metrics often fall short of capturing the true targets of recommendation tasks, leading to suboptimal performance. To overcome this dilemma, we propose a new optimization metric, Lower-Left Partial AUC (LLPAUC), which is computationally efficient like AUC but strongly correlates with Top-K ranking metrics. Compared to AUC, LLPAUC considers only the partial area under the ROC curve in the Lower-Left corner to push the optimization focus on Top-K. We provide theoretical validation of the correlation between LLPAUC and Top-K ranking metrics and demonstrate its robustness to noisy user feedback. We further design an efficient point-wise recommendation loss to maximize LLPAUC and evaluate it on three datasets, validating its effectiveness and robustness.

Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation

TL;DR

This work tackles the gap between efficient optimization metrics and Top-K–oriented evaluation in recommender systems. It introduces Lower-Left Partial AUC (LLPAUC), a constrained partial AUC that focuses on the Lower-Left ROC region to prioritize top-ranked items while retaining AUC-like efficiency. The authors provide theoretical results showing LLPAUC's tighter correlation with Recall@K and Precision@K than AUC or OPAUC, and they develop a differentiable, minimax, point-wise loss using an Average Top-K trick to maximize LLPAUC efficiently. Extensive experiments on multiple real-world datasets demonstrate LLPAUC’s superior performance and robustness to noisy feedback across different backbones, with convergence comparable to existing losses. The work offers a practical, scalable optimization objective that aligns closely with Top-K goals, enabling better large-scale recommendations in real-world systems.

Abstract

Optimization metrics are crucial for building recommendation systems at scale. However, an effective and efficient metric for practical use remains elusive. While Top-K ranking metrics are the gold standard for optimization, they suffer from significant computational overhead. Alternatively, the more efficient accuracy and AUC metrics often fall short of capturing the true targets of recommendation tasks, leading to suboptimal performance. To overcome this dilemma, we propose a new optimization metric, Lower-Left Partial AUC (LLPAUC), which is computationally efficient like AUC but strongly correlates with Top-K ranking metrics. Compared to AUC, LLPAUC considers only the partial area under the ROC curve in the Lower-Left corner to push the optimization focus on Top-K. We provide theoretical validation of the correlation between LLPAUC and Top-K ranking metrics and demonstrate its robustness to noisy user feedback. We further design an efficient point-wise recommendation loss to maximize LLPAUC and evaluate it on three datasets, validating its effectiveness and robustness.
Paper Structure (29 sections, 6 theorems, 23 equations, 7 figures, 5 tables, 1 algorithm)

This paper contains 29 sections, 6 theorems, 23 equations, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Task Formulation
  5. AUC And Partial AUC
  6. When LLPAUC Meets with Recommender System
  7. Theoretical Analysis
  8. Empirical Analysis
  9. Method
  10. Loss Function
  11. Experiments
  12. Experiments Setting
  13. Main Results
  14. In-depth Analysis
  15. Conclusion and Future Work
  16. ...and 14 more sections

Key Result

Theorem 1

Suppose there are $n^+$ positive items and $n^-$ negative items, where $n^+>K$ and $n^->K$. Ranking all items in descending order according to the prediction scores obtained from any model f, we have where $\alpha =\frac{K}{n^+}$, $\beta=\frac{K}{n^-}$, and

Figures (7)

  • Figure 1: (a) AUC measures the entire area under the ROC curve; (b) LLPAUC considers the lower-left corner; (c) Compared to AUC, LLPAUC only considers the ranking for top-ranked items.
  • Figure 2: Pearson correlation coefficient between Recall@K and LLPAUC$(\alpha,\beta)$.
  • Figure 3: Ablation studies among different AUC metrics with clean training and noise training.
  • Figure 4: Normalized Recall@K on Adressa dataset under clean training for K=20, 50 and 100.
  • Figure 5: Given a fix $\beta$, the hyperparameter analysis of $\alpha$ in LLPAUC$(\alpha, \beta)$ on different datasets under clean training setting and noise training setting.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4