Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?

TL;DR

The MovieLens platform demonstrates an efficient and effective way of collecting user preferences to address cold-starts and models that achieve excellent recommendation accuracy on the MovieLens dataset may not demonstrate superior performance in practice, for at least two kinds of differences: the differences in the contexts of user-item interaction generation, and the differences in user knowledge about the item collections.

Abstract

A typical benchmark dataset for recommender system (RecSys) evaluation consists of user-item interactions generated on a platform within a time period. The interaction generation mechanism partially explains why a user interacts with (e.g., like, purchase, rate) an item, and the context of when a particular interaction happened. In this study, we conduct a meticulous analysis of the MovieLens dataset and explain the potential impact of using the dataset for evaluating recommendation algorithms. We make a few main findings from our analysis. First, there are significant differences in user interactions at the different stages when a user interacts with the MovieLens platform. The early interactions largely define the user portrait which affects the subsequent interactions. Second, user interactions are highly affected by the candidate movies that are recommended by the platform's internal recommendation algorithm(s). Third, changing the order of user interactions makes it more difficult for sequential algorithms to capture the progressive interaction process. We further discuss the discrepancy between the interaction generation mechanism that is employed by the MovieLens system and that of typical real-world recommendation scenarios. In summary, the MovieLens platform demonstrates an efficient and effective way of collecting user preferences to address cold-starts. However, models that achieve excellent recommendation accuracy on the MovieLens dataset may not demonstrate superior performance in practice, for at least two kinds of differences: (i) the differences in the contexts of user-item interaction generation, and (ii) the differences in user knowledge about the item collections. While results on MovieLens can be useful as a reference, they should not be solely relied upon as the primary justification for the effectiveness of a recommendation system model.

Figures (8)

• Figure 1: Illustration of the three stages ($R_0$, $R_1$, and $R_2$) when a user interacts with MovieLens. From $R_0$ to $R_3$, the search space of candidate movies becomes larger and the movies recommended become more personalized, from a common root to various branches in a tree representation (the figure on the left-hand side). In the figure on the right-hand side, $M_1$ and $M_2$ represent the two pools of candidate movies to generate the list of recommended movies for users to rate, in stages $R_1$ and $R_2$ respectively. $P_0$ is the group-based user preference. $P_1$ and $P_2$ are the item-based preferences (i.e., movie ratings) by different internal recommendation algorithms on MovieLens with candidate pools $M_1$ and $M_2$ respectively.
• Figure 2: Screencaptures on MovieLens website for the three stages: (a) for stage $R_0$, (b) for stages $R_1$ and $R_2$.
• Figure 3: The HR@10 results of removing 15 interactions from the training set of each user, on four yearly datasets, and the entire MovieLens1518 dataset covering all four years (indicated by "4 Yrs"). We evaluate three cases: (i) removal of the first 15 interactions, (ii) removal of the last 15 interactions, and (iii) removal of randomly sampled 15 interactions. The 95% confidence intervals are indicated for the performance of non-deterministic algorithms.
• Figure 4: The NDCG@10 results of removing 15 interactions from the training set of each user, on four yearly datasets, and the entire MovieLens1518 dataset covering all four years (indicated by "4 Yrs"). We evaluate three cases: (i) removal of the first 15 interactions, (ii) removal of the last 15 interactions, and (iii) removal of randomly sampled 15 interactions. The 95% confidence intervals are indicated for the performance of non-deterministic algorithms.
• Figure 5: Comparison of the size of the candidate movie pool at different stages, on four yearly subsets, and also the entire MovieLens1518 dataset covering "4 Yrs".