Revealing the Hidden Impact of Top-N Metrics on Optimization in Recommender Systems

TL;DR

This paper investigates whether selecting items beyond the top-$n$ during evaluation can improve predictive accuracy in top-$n$ recommender systems, addressing a potential misalignment between training losses and optimization-target metrics. By exhaustively evaluating 252 selection strategies (choosing $n$ items from the top-$|K|$ items with $|K|=10$ and $n=5$) across 12 implicit and 8 explicit datasets and 11 algorithms, the study shows that while non-top-$n$ strategies can outperform top-$n$ in some cases, the gains are negligible on average. Statistical analyses indicate most strategies are not significantly different, and high generalization from validation to test supports reliable conclusions. The findings suggest there is no practical need to optimize top-$n$ selection strategies, reinforcing the validity of standard evaluation practice and contributing to reproducibility; the work also highlights domain- and algorithm-dependent nuances that warrant awareness. The authors provide public code to reproduce the experiments and encourage future work on efficient search methods for near-top selections.

Abstract

The hyperparameters of recommender systems for top-n predictions are typically optimized to enhance the predictive performance of algorithms. Thereby, the optimization algorithm, e.g., grid search or random search, searches for the best hyperparameter configuration according to an optimization-target metric, like nDCG or Precision. In contrast, the optimized algorithm, internally optimizes a different loss function during training, like squared error or cross-entropy. To tackle this discrepancy, recent work focused on generating loss functions better suited for recommender systems. Yet, when evaluating an algorithm using a top-n metric during optimization, another discrepancy between the optimization-target metric and the training loss has so far been ignored. During optimization, the top-n items are selected for computing a top-n metric; ignoring that the top-n items are selected from the recommendations of a model trained with an entirely different loss function. Item recommendations suitable for optimization-target metrics could be outside the top-n recommended items; hiddenly impacting the optimization performance. Therefore, we were motivated to analyze whether the top-n items are optimal for optimization-target top-n metrics. In pursuit of an answer, we exhaustively evaluate the predictive performance of 250 selection strategies besides selecting the top-n. We extensively evaluate each selection strategy over twelve implicit feedback and eight explicit feedback data sets with eleven recommender systems algorithms. Our results show that there exist selection strategies other than top-n that increase predictive performance for various algorithms and recommendation domains. However, the performance of the top ~43% of selection strategies is not significantly different. We discuss the impact of our findings on optimization and re-ranking in recommender systems and feasible solutions.

Figures (4)

• Figure 1: The relative nDCG performance of the best non-top-n selection strategy versus the top-n selection strategy evaluated on the test set. A point to the right of the red line indicates that the best non-top-n selection strategy is better than the top-n selection strategy.
• Figure 2: This Figure is a focused view on evaluating data sets that represent the articles (\ref{['articles_domain']}) and movies (\ref{['movies_domain']}) recommendation domains. The shown data points are a subset of Figures \ref{['implicit_performance']} and \ref{['explicit_performance']}, respectively, but with a focus on the data sets. The data sets are indicated with markers and colors to distinguish them better.
• Figure 3: The exhaustive evaluation of 252 selection strategies with the nDCG metric on the data set Adressa One Week with the Logistic Matrix Factorization algorithm (\ref{['adressa']}) and the data set MovieLens-100k with the Alternating Least Squares algorithm (\ref{['ml100k']}). Each selection strategy picks a different combination of 5 items from the top 10 predicted items. The performance of a selection strategy is averaged over all users in the test data set. The y-axis refers to the index of the items in the predicted ranked list, e.g., element 0 is the highest-ranked and the most relevant item. A black dot refers to a selection strategy that contains said item, e.g., all selection strategies represented by black dots in a row contain the item according to the index stated on the y-axis.
• Figure 4: The relative performance of selection strategies on the validation and test set for the data sets Adressa One Week (\ref{['adressa_generalization']}) and MovieLens-100k (\ref{['ml100k_generalization']}). If a point is on the identity line, the validation and test set have equal relative performance and, therefore, generalize.