Can Large Language Models Understand Preferences in Personalized Recommendation?

TL;DR

The paper tackles the challenge that traditional rating-based metrics may misrepresent an LLM’s ability to personalize recommendations due to user rating bias and item quality. It introduces PerRecBench, a grouped-ranking benchmark that isolates true personalization signals via relative ratings and observed preferences, evaluating pointwise, pairwise, and listwise prompting with Kendall’s tau as the metric. Across 19 LLMs, larger models show limits in learning personalized preferences, with pairwise/listwise ranking outperforming pointwise and open-source models rivaling proprietary ones, while prompting strategies and pretraining-domain distributions meaningfully affect performance. The work also investigates fine-tuning strategies, finding weight merging to be the most effective among those studied, yet concluding that robust LLM-based personalization remains an open problem. The benchmark and findings underscore the need for designing personalization-aware data, prompts, and training regimes to advance practical, fair, and privacy-conscious personalized recommendations.

Abstract

Large Language Models (LLMs) excel in various tasks, including personalized recommendations. Existing evaluation methods often focus on rating prediction, relying on regression errors between actual and predicted ratings. However, user rating bias and item quality, two influential factors behind rating scores, can obscure personal preferences in user-item pair data. To address this, we introduce PerRecBench, disassociating the evaluation from these two factors and assessing recommendation techniques on capturing the personal preferences in a grouped ranking manner. We find that the LLM-based recommendation techniques that are generally good at rating prediction fail to identify users' favored and disfavored items when the user rating bias and item quality are eliminated by grouping users. With PerRecBench and 19 LLMs, we find that while larger models generally outperform smaller ones, they still struggle with personalized recommendation. Our findings reveal the superiority of pairwise and listwise ranking approaches over pointwise ranking, PerRecBench's low correlation with traditional regression metrics, the importance of user profiles, and the role of pretraining data distributions. We further explore three supervised fine-tuning strategies, finding that merging weights from single-format training is promising but improving LLMs' understanding of user preferences remains an open research problem. Code and data are available at https://github.com/TamSiuhin/PerRecBench

Figures (8)

• Figure 1: MAE and RMSE performance of user rating bias (average user rating history), item quality (average item rating), and existing LLM-based personalization methods. Simple averages of user rating history and item quality, which do not consider individual preferences, achieve state-of-the-art performance across four shopping domains, questioning the validity of MAE and RMSE for evaluating personalization.
• Figure 2: Overview of PerRecBench, where the LLM ranks user preferences for a query item using pointwise, pairwise, and listwise prompting. The ground-truth ranking is derived from relative ratings, calculated as the user’s actual rating minus their average rating, to mitigate user rating bias. Finally, Kendall’s tau is computed between the predicted ranking and the ground-truth ranking to evaluate performance.
• Figure 3: Correlation between Kendall’s tau in PerRecBench and traditional metrics (MAE and RMSE). The results show weak correlation, further confirming that MAE and RMSE are not reliable indicators of personalization capabilities.
• Figure 4: Performance across different domains and average item name count in the pretraining dataset. Query items with higher frequency in pretraining data generally show better performance in PerRecBench.
• Figure 5: The statistics of PerRecBench, including the distribution of actual rating, relative rating, length of user history, and the purchase date.