Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

User Preferences for Large Language Model versus Template-Based Explanations of Movie Recommendations: A Pilot Study

Julien Albert, Martin Balfroid, Miriam Doh, Jeremie Bogaert, Luca La Fisca, Liesbet De Vos, Bryan Renard, Vincent Stragier, Emmanuel Jean

TL;DR

Although subject to high variance, preliminary findings suggest that LLM-based explanations may provide a richer and more engaging user experience, further aligning with user expectations.

Abstract

Recommender systems have become integral to our digital experiences, from online shopping to streaming platforms. Still, the rationale behind their suggestions often remains opaque to users. While some systems employ a graph-based approach, offering inherent explainability through paths associating recommended items and seed items, non-experts could not easily understand these explanations. A popular alternative is to convert graph-based explanations into textual ones using a template and an algorithm, which we denote here as ''template-based'' explanations. Yet, these can sometimes come across as impersonal or uninspiring. A novel method would be to employ large language models (LLMs) for this purpose, which we denote as ''LLM-based''. To assess the effectiveness of LLMs in generating more resonant explanations, we conducted a pilot study with 25 participants. They were presented with three explanations: (1) traditional template-based, (2) LLM-based rephrasing of the template output, and (3) purely LLM-based explanations derived from the graph-based explanations. Although subject to high variance, preliminary findings suggest that LLM-based explanations may provide a richer and more engaging user experience, further aligning with user expectations. This study sheds light on the potential limitations of current explanation methods and offers promising directions for leveraging large language models to improve user satisfaction and trust in recommender systems.

User Preferences for Large Language Model versus Template-Based Explanations of Movie Recommendations: A Pilot Study

TL;DR

Although subject to high variance, preliminary findings suggest that LLM-based explanations may provide a richer and more engaging user experience, further aligning with user expectations.

Abstract

Recommender systems have become integral to our digital experiences, from online shopping to streaming platforms. Still, the rationale behind their suggestions often remains opaque to users. While some systems employ a graph-based approach, offering inherent explainability through paths associating recommended items and seed items, non-experts could not easily understand these explanations. A popular alternative is to convert graph-based explanations into textual ones using a template and an algorithm, which we denote here as ''template-based'' explanations. Yet, these can sometimes come across as impersonal or uninspiring. A novel method would be to employ large language models (LLMs) for this purpose, which we denote as ''LLM-based''. To assess the effectiveness of LLMs in generating more resonant explanations, we conducted a pilot study with 25 participants. They were presented with three explanations: (1) traditional template-based, (2) LLM-based rephrasing of the template output, and (3) purely LLM-based explanations derived from the graph-based explanations. Although subject to high variance, preliminary findings suggest that LLM-based explanations may provide a richer and more engaging user experience, further aligning with user expectations. This study sheds light on the potential limitations of current explanation methods and offers promising directions for leveraging large language models to improve user satisfaction and trust in recommender systems.
Paper Structure (17 sections, 5 figures)

This paper contains 17 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Technical Exploration and Implementation
  3. User-based evaluation
  4. Methodology
  5. Results
  6. Future Works
  7. Authors' Biographies
  8. Julien Albert
  9. Martin Balfroid
  10. Miriam Doh
  11. Jeremie Bogaert
  12. Luca La Fisca
  13. Liesbet De Vos
  14. Bryan Renard
  15. Vincent Stragier
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Pipeline used to guide our experiments. Methods and models used for the evaluation part are in fuchsia.
  • Figure 2: Illustration of the three types of explanations compared in the user evaluation.
  • Figure 3: Here is an example of the same recommendation presented in the same format as the prompt in Liu et al. liu2023chatgpt. Black-colored text outlines the task, red-colored text highlights the formatting guidelines, and blue-colored text is either the given template or the graph.
  • Figure 4: Structure of our user evaluation procedure
  • Figure 5: User assessment of explanations w.r.t. the 7 goals. about the recommendation explanations.