Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CROLoss: Towards a Customizable Loss for Retrieval Models in Recommender Systems

Yongxiang Tang, Wentao Bai, Guilin Li, Xialong Liu, Yu Zhang

TL;DR

The proposed CROLoss formulation defines a more generalized loss function space, covering most of the conventional loss functions as special cases, and develops the Lambda method, a gradient-based method that invites more flexibility and can further boost the system performance.

Abstract

In large-scale recommender systems, retrieving top N relevant candidates accurately with resource constrain is crucial. To evaluate the performance of such retrieval models, Recall@N, the frequency of positive samples being retrieved in the top N ranking, is widely used. However, most of the conventional loss functions for retrieval models such as softmax cross-entropy and pairwise comparison methods do not directly optimize Recall@N. Moreover, those conventional loss functions cannot be customized for the specific retrieval size N required by each application and thus may lead to sub-optimal performance. In this paper, we proposed the Customizable Recall@N Optimization Loss (CROLoss), a loss function that can directly optimize the Recall@N metrics and is customizable for different choices of N. This proposed CROLoss formulation defines a more generalized loss function space, covering most of the conventional loss functions as special cases. Furthermore, we develop the Lambda method, a gradient-based method that invites more flexibility and can further boost the system performance. We evaluate the proposed CROLoss on two public benchmark datasets. The results show that CROLoss achieves SOTA results over conventional loss functions for both datasets with various choices of retrieval size N. CROLoss has been deployed onto our online E-commerce advertising platform, where a fourteen-day online A/B test demonstrated that CROLoss contributes to a significant business revenue growth of 4.75%.

CROLoss: Towards a Customizable Loss for Retrieval Models in Recommender Systems

TL;DR

The proposed CROLoss formulation defines a more generalized loss function space, covering most of the conventional loss functions as special cases, and develops the Lambda method, a gradient-based method that invites more flexibility and can further boost the system performance.

Abstract

In large-scale recommender systems, retrieving top N relevant candidates accurately with resource constrain is crucial. To evaluate the performance of such retrieval models, Recall@N, the frequency of positive samples being retrieved in the top N ranking, is widely used. However, most of the conventional loss functions for retrieval models such as softmax cross-entropy and pairwise comparison methods do not directly optimize Recall@N. Moreover, those conventional loss functions cannot be customized for the specific retrieval size N required by each application and thus may lead to sub-optimal performance. In this paper, we proposed the Customizable Recall@N Optimization Loss (CROLoss), a loss function that can directly optimize the Recall@N metrics and is customizable for different choices of N. This proposed CROLoss formulation defines a more generalized loss function space, covering most of the conventional loss functions as special cases. Furthermore, we develop the Lambda method, a gradient-based method that invites more flexibility and can further boost the system performance. We evaluate the proposed CROLoss on two public benchmark datasets. The results show that CROLoss achieves SOTA results over conventional loss functions for both datasets with various choices of retrieval size N. CROLoss has been deployed onto our online E-commerce advertising platform, where a fourteen-day online A/B test demonstrated that CROLoss contributes to a significant business revenue growth of 4.75%.
Paper Structure (19 sections, 24 equations, 5 figures, 4 tables)

This paper contains 19 sections, 24 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval Model Optimization Losses
  4. Learning to Rank
  5. Customizable Recall@N Optimization Loss
  6. The base model
  7. Problem Formulation
  8. Customizable Retrieval Range
  9. Pairwise Kernel-Based Objective
  10. Relation to Existing Methods
  11. The Lambda Method
  12. Experiment
  13. Experimental Setup
  14. Performance Comparison (RQ1)
  15. Influence of Kernel and Alpha (RQ2)
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Architecture of two tower retrieval model
  • Figure 2: Example of optional comparison kernel functions.
  • Figure 3: The Recall@N metrics of four different comparison kernels of CROLoss at different retrieval size $N$ on Amazon dataset.
  • Figure 4: The Recall@N metrics of CROLoss with sigmoid kernel and different weighting parameter $\alpha$ on the Amazon dataset.
  • Figure 5: Performance improvement of using CROLoss for hard negative mining on triplet loss and BPR loss.