Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Diversity, Novelty, and Popularity Bias in ChatGPT's Recommendations

Dario Di Palma, Giovanni Maria Biancofiore, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia

TL;DR

This paper investigates the beyond-accuracy performance of ChatGPT based recommendations, focusing on diversity, novelty, and popularity bias across three domains (Books, Movies, Music) and in cold-start scenarios. It evaluates both ChatGPT-3.5 and GPT-4 using hand crafted prompts, with Role-Playing prompting identified as the most effective approach, and compares against strong non-ChatGPT baselines. Across three datasets, GPT-4 generally matches or surpasses traditional recommenders in accuracy and demonstrates balanced diversity and novelty, with notable strengths in cold-start settings. The study highlights both the potential and limitations of ChatGPT as a recommender, including concerns about domain variability and memorization, and proposes directions for broader evaluation and comparisons with other large language models.

Abstract

ChatGPT has emerged as a versatile tool, demonstrating capabilities across diverse domains. Given these successes, the Recommender Systems (RSs) community has begun investigating its applications within recommendation scenarios primarily focusing on accuracy. While the integration of ChatGPT into RSs has garnered significant attention, a comprehensive analysis of its performance across various dimensions remains largely unexplored. Specifically, the capabilities of providing diverse and novel recommendations or exploring potential biases such as popularity bias have not been thoroughly examined. As the use of these models continues to expand, understanding these aspects is crucial for enhancing user satisfaction and achieving long-term personalization. This study investigates the recommendations provided by ChatGPT-3.5 and ChatGPT-4 by assessing ChatGPT's capabilities in terms of diversity, novelty, and popularity bias. We evaluate these models on three distinct datasets and assess their performance in Top-N recommendation and cold-start scenarios. The findings reveal that ChatGPT-4 matches or surpasses traditional recommenders, demonstrating the ability to balance novelty and diversity in recommendations. Furthermore, in the cold-start scenario, ChatGPT models exhibit superior performance in both accuracy and novelty, suggesting they can be particularly beneficial for new users. This research highlights the strengths and limitations of ChatGPT's recommendations, offering new perspectives on the capacity of these models to provide recommendations beyond accuracy-focused metrics.

Exploring Diversity, Novelty, and Popularity Bias in ChatGPT's Recommendations

TL;DR

This paper investigates the beyond-accuracy performance of ChatGPT based recommendations, focusing on diversity, novelty, and popularity bias across three domains (Books, Movies, Music) and in cold-start scenarios. It evaluates both ChatGPT-3.5 and GPT-4 using hand crafted prompts, with Role-Playing prompting identified as the most effective approach, and compares against strong non-ChatGPT baselines. Across three datasets, GPT-4 generally matches or surpasses traditional recommenders in accuracy and demonstrates balanced diversity and novelty, with notable strengths in cold-start settings. The study highlights both the potential and limitations of ChatGPT as a recommender, including concerns about domain variability and memorization, and proposes directions for broader evaluation and comparisons with other large language models.

Abstract

ChatGPT has emerged as a versatile tool, demonstrating capabilities across diverse domains. Given these successes, the Recommender Systems (RSs) community has begun investigating its applications within recommendation scenarios primarily focusing on accuracy. While the integration of ChatGPT into RSs has garnered significant attention, a comprehensive analysis of its performance across various dimensions remains largely unexplored. Specifically, the capabilities of providing diverse and novel recommendations or exploring potential biases such as popularity bias have not been thoroughly examined. As the use of these models continues to expand, understanding these aspects is crucial for enhancing user satisfaction and achieving long-term personalization. This study investigates the recommendations provided by ChatGPT-3.5 and ChatGPT-4 by assessing ChatGPT's capabilities in terms of diversity, novelty, and popularity bias. We evaluate these models on three distinct datasets and assess their performance in Top-N recommendation and cold-start scenarios. The findings reveal that ChatGPT-4 matches or surpasses traditional recommenders, demonstrating the ability to balance novelty and diversity in recommendations. Furthermore, in the cold-start scenario, ChatGPT models exhibit superior performance in both accuracy and novelty, suggesting they can be particularly beneficial for new users. This research highlights the strengths and limitations of ChatGPT's recommendations, offering new perspectives on the capacity of these models to provide recommendations beyond accuracy-focused metrics.
Paper Structure (9 sections, 4 figures, 4 tables)

This paper contains 9 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Prompt Design
  5. Experimental Setup
  6. Experimental Results
  7. ChatGPT Beyond-Accuracy Recommendation Performance
  8. User Cold-Start Scenario
  9. Conclusion

Figures (4)

  • Figure 1: Example of a zero-shot prompt designed to generate recommendations based on the user's history.
  • Figure 2: Example of a few-shot prompt illustrating recommendations with explanations based on the user's watched movies and ratings.
  • Figure 3: Example of a Chain-of-Thought prompt for book recommendation, incorporating user preferences and step-by-step reasoning to arrive at a recommendation.
  • Figure 4: Example of a Role-Playing Recommender prompt designed to generate a ranked list of 50 recommendations based on the user's history.