Beyond Utility: Evaluating LLM as Recommender

TL;DR

A multidimensional evaluation framework is proposed and it is found that LLMs excel at handling tasks with prior knowledge and shorter input histories in the ranking setting, and perform better in the re-ranking setting, beating traditional models across multiple dimensions.

Abstract

With the rapid development of Large Language Models (LLMs), recent studies employed LLMs as recommenders to provide personalized information services for distinct users. Despite efforts to improve the accuracy of LLM-based recommendation models, relatively little attention is paid to beyond-utility dimensions. Moreover, there are unique evaluation aspects of LLM-based recommendation models, which have been largely ignored. To bridge this gap, we explore four new evaluation dimensions and propose a multidimensional evaluation framework. The new evaluation dimensions include: 1) history length sensitivity, 2) candidate position bias, 3) generation-involved performance, and 4) hallucinations. All four dimensions have the potential to impact performance, but are largely unnecessary for consideration in traditional systems. Using this multidimensional evaluation framework, along with traditional aspects, we evaluate the performance of seven LLM-based recommenders, with three prompting strategies, comparing them with six traditional models on both ranking and re-ranking tasks on four datasets. We find that LLMs excel at handling tasks with prior knowledge and shorter input histories in the ranking setting, and perform better in the re-ranking setting, beating traditional models across multiple dimensions. However, LLMs exhibit substantial candidate position bias issues, and some models hallucinate non-existent items much more often than others. We intend our evaluation framework and observations to benefit future research on the use of LLMs as recommenders. The code and data are available at https://github.com/JiangDeccc/EvaLLMasRecommender.

Figures (7)

• Figure 1: Multidimensional evaluation process of LLM as recommender.
• Figure 2: Novelty Performances in Beauty and LastFM. The figures display the top 10 best-performing methods.
• Figure 3: History Length Sensitivity of the best traditional models (blue dashed line) and LLM-powered methods (yellow & red solid line).
• Figure 4: Candidate Position Bias of LLM-based methods: recommendation accuracy when positive items placed at different positions within the candidate list.
• Figure 5: An example of LLM-Generated Profile.