Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recommender Systems Algorithm Selection for Ranking Prediction on Implicit Feedback Datasets

Lukas Wegmeth, Tobias Vente, Joeran Beel

TL;DR

It is demonstrated that the best optimized traditional meta-model, e.g., XGBoost, achieves a recall of 48.6%, outperforming the best tested automated machine learning meta-model, e.g., AutoGluon, which achieves a recall of 47.2%.

Abstract

The recommender systems algorithm selection problem for ranking prediction on implicit feedback datasets is under-explored. Traditional approaches in recommender systems algorithm selection focus predominantly on rating prediction on explicit feedback datasets, leaving a research gap for ranking prediction on implicit feedback datasets. Algorithm selection is a critical challenge for nearly every practitioner in recommender systems. In this work, we take the first steps toward addressing this research gap. We evaluate the NDCG@10 of 24 recommender systems algorithms, each with two hyperparameter configurations, on 72 recommender systems datasets. We train four optimized machine-learning meta-models and one automated machine-learning meta-model with three different settings on the resulting meta-dataset. Our results show that the predictions of all tested meta-models exhibit a median Spearman correlation ranging from 0.857 to 0.918 with the ground truth. We show that the median Spearman correlation between meta-model predictions and the ground truth increases by an average of 0.124 when the meta-model is optimized to predict the ranking of algorithms instead of their performance. Furthermore, in terms of predicting the best algorithm for an unknown dataset, we demonstrate that the best optimized traditional meta-model, e.g., XGBoost, achieves a recall of 48.6%, outperforming the best tested automated machine learning meta-model, e.g., AutoGluon, which achieves a recall of 47.2%.

Recommender Systems Algorithm Selection for Ranking Prediction on Implicit Feedback Datasets

TL;DR

It is demonstrated that the best optimized traditional meta-model, e.g., XGBoost, achieves a recall of 48.6%, outperforming the best tested automated machine learning meta-model, e.g., AutoGluon, which achieves a recall of 47.2%.

Abstract

The recommender systems algorithm selection problem for ranking prediction on implicit feedback datasets is under-explored. Traditional approaches in recommender systems algorithm selection focus predominantly on rating prediction on explicit feedback datasets, leaving a research gap for ranking prediction on implicit feedback datasets. Algorithm selection is a critical challenge for nearly every practitioner in recommender systems. In this work, we take the first steps toward addressing this research gap. We evaluate the NDCG@10 of 24 recommender systems algorithms, each with two hyperparameter configurations, on 72 recommender systems datasets. We train four optimized machine-learning meta-models and one automated machine-learning meta-model with three different settings on the resulting meta-dataset. Our results show that the predictions of all tested meta-models exhibit a median Spearman correlation ranging from 0.857 to 0.918 with the ground truth. We show that the median Spearman correlation between meta-model predictions and the ground truth increases by an average of 0.124 when the meta-model is optimized to predict the ranking of algorithms instead of their performance. Furthermore, in terms of predicting the best algorithm for an unknown dataset, we demonstrate that the best optimized traditional meta-model, e.g., XGBoost, achieves a recall of 48.6%, outperforming the best tested automated machine learning meta-model, e.g., AutoGluon, which achieves a recall of 47.2%.
Paper Structure (9 sections, 2 figures)

This paper contains 9 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Dataset Processing
  5. Meta-Features
  6. Algorithms
  7. Meta-Learner
  8. Results
  9. Discussion

Figures (2)

  • Figure 1: The Spearman correlation between meta-model predictions and ground truth (NDCG@10) per dataset. Each data point represents the correlation between predicted rankings (first plot) or performance (second plot) and the ground truth for a test dataset in a leave-one-out evaluation.
  • Figure 2: The Recall@1 and Recall@3 for the ranking objective meta-model predictions show the frequency of achieving the specified recall per dataset in a leave-one-out evaluation. For example, a Recall@1 score of 1 means the meta-model correctly identified the top algorithm. Each meta-model is evaluated on 72 datasets.