Can Large Language Models Assess Serendipity in Recommender Systems?
Yu Tokutake, Kazushi Okamoto
TL;DR
This work investigates whether large language models (LLMs) can reliably assess serendipity in recommender systems. It formalizes a serendipity-detection task using LLM prompts that incorporate a user's rating history and item metadata, and evaluates several LLMs against ground-truth serendipity labels from the Serendipity-2018 dataset, comparing to traditional and serendipity-aware baselines. Results show that while LLMs can outperform some baselines and that including item genres improves performance, their agreement with human judgments is not strong, and their outputs are not fully interpretable via standard serendipity metrics. The study highlights the potential of LLMs for serendipity assessment and proposes directions for prompt refinement, interpretability, and data-augmentation to enhance serendipity-aware recommendations in practice.
Abstract
Serendipity-oriented recommender systems aim to counteract over-specialization in user preferences. However, evaluating a user's serendipitous response towards a recommended item can be challenging because of its emotional nature. In this study, we address this issue by leveraging the rich knowledge of large language models (LLMs), which can perform a variety of tasks. First, this study explored the alignment between serendipitous evaluations made by LLMs and those made by humans. In this investigation, a binary classification task was given to the LLMs to predict whether a user would find the recommended item serendipitously. The predictive performances of three LLMs on a benchmark dataset in which humans assigned the ground truth of serendipitous items were measured. The experimental findings reveal that LLM-based assessment methods did not have a very high agreement rate with human assessments. However, they performed as well as or better than the baseline methods. Further validation results indicate that the number of user rating histories provided to LLM prompts should be carefully chosen to avoid both insufficient and excessive inputs and that the output of LLMs that show high classification performance is difficult to interpret.