Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating User Experience of ChatGPT-based Conversational Recommender Systems: The Effects of Prompt Guidance and Recommendation Domain

Yizhe Zhang, Yucheng Jin, Li Chen, Ting Yang

TL;DR

The paper addresses how prompt guidance and recommendation domain influence the user experience of ChatGPT-based conversational recommender systems. It employs a mixed-method online study (N=100) with a 2x2 design (PG: with/without; RD: book/job) to evaluate UX across multiple CRS-Que dimensions and interaction metrics. Key findings show that PG enhances explainability, ease of use, transparency, and CUI adaptability, while RD modulates novelty and willingness to use/try, with domain stakes shaping PG's effectiveness. The results offer design guidance for domain-aware, user-tailored prompt strategies in ChatGPT-powered CRS and highlight moderation by prior RS experience, informing practical UX design and future research.

Abstract

Conversational recommender systems (CRS) enable users to articulate their preferences and provide feedback through natural language. With the advent of large language models (LLMs), the potential to enhance user engagement with CRS and augment the recommendation process with LLM-generated content has received increasing attention. However, the efficacy of LLM-powered CRS is contingent upon the use of prompts, and the subjective perception of recommendation quality can differ across various recommendation domains. Therefore, we have developed a ChatGPT-based CRS to investigate the impact of these two factors, prompt guidance (PG) and recommendation domain (RD), on the overall user experience of the system. We conducted an online empirical study (N = 100) by employing a mixed-method approach that utilized a between-subjects design for the variable of PG (with vs. without) and a within-subjects design for RD (book recommendations vs. job recommendations). The findings reveal that PG can substantially enhance the system's explainability, adaptability, perceived ease of use, and transparency. Moreover, users are inclined to perceive a greater sense of novelty and demonstrate a higher propensity to engage with and try recommended items in the context of book recommendations as opposed to job recommendations. Furthermore, the influence of PG on certain user experience metrics and interactive behaviors appears to be modulated by the recommendation domain, as evidenced by the interaction effects between the two examined factors. This work contributes to the user-centered evaluation of ChatGPT-based CRS by investigating two prominent factors and offers practical design guidance.

Navigating User Experience of ChatGPT-based Conversational Recommender Systems: The Effects of Prompt Guidance and Recommendation Domain

TL;DR

The paper addresses how prompt guidance and recommendation domain influence the user experience of ChatGPT-based conversational recommender systems. It employs a mixed-method online study (N=100) with a 2x2 design (PG: with/without; RD: book/job) to evaluate UX across multiple CRS-Que dimensions and interaction metrics. Key findings show that PG enhances explainability, ease of use, transparency, and CUI adaptability, while RD modulates novelty and willingness to use/try, with domain stakes shaping PG's effectiveness. The results offer design guidance for domain-aware, user-tailored prompt strategies in ChatGPT-powered CRS and highlight moderation by prior RS experience, informing practical UX design and future research.

Abstract

Conversational recommender systems (CRS) enable users to articulate their preferences and provide feedback through natural language. With the advent of large language models (LLMs), the potential to enhance user engagement with CRS and augment the recommendation process with LLM-generated content has received increasing attention. However, the efficacy of LLM-powered CRS is contingent upon the use of prompts, and the subjective perception of recommendation quality can differ across various recommendation domains. Therefore, we have developed a ChatGPT-based CRS to investigate the impact of these two factors, prompt guidance (PG) and recommendation domain (RD), on the overall user experience of the system. We conducted an online empirical study (N = 100) by employing a mixed-method approach that utilized a between-subjects design for the variable of PG (with vs. without) and a within-subjects design for RD (book recommendations vs. job recommendations). The findings reveal that PG can substantially enhance the system's explainability, adaptability, perceived ease of use, and transparency. Moreover, users are inclined to perceive a greater sense of novelty and demonstrate a higher propensity to engage with and try recommended items in the context of book recommendations as opposed to job recommendations. Furthermore, the influence of PG on certain user experience metrics and interactive behaviors appears to be modulated by the recommendation domain, as evidenced by the interaction effects between the two examined factors. This work contributes to the user-centered evaluation of ChatGPT-based CRS by investigating two prominent factors and offers practical design guidance.
Paper Structure (27 sections, 5 figures, 4 tables)

This paper contains 27 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. ChatGPT-based CRS
  4. User Experience of CRS
  5. Prompt Guidance and Recommendation Domains
  6. Experimental setup
  7. Procedure
  8. Participants
  9. Design Manipulations
  10. Prompt Guidance (PG)
  11. Recommendation Domain (RD)
  12. User Interfaces
  13. Measurements
  14. Pre-study Questionnaire
  15. Post-study Questionnaire
  16. ...and 12 more sections

Figures (5)

  • Figure 1: The ChatGPT-based CRS with prompt guidance (left) and without prompt guidance (right) for book recommendations.
  • Figure 2: The summary of effects of prompt guidance (PG) and recommendation domain (RD)
  • Figure 3: Results of the UX measurements of CRS grouped by the conditions of two independent variables: RD and PG. A cut-off value of 3.5 represents agreement on the five-point Likert scale. * is marked for significant difference at the 5% level (p-value < 0.05).
  • Figure 4: Interaction Effects of PG and RD on Accuracy (top-left), User Control (bottom-left), CUI Adaptability (top-right), and the Average Words per Conversation (bottom-right).
  • Figure 5: Moderation effects of the personal characteristic "experience with recommender systems" on UX aspects explainability (left), perceived ease of use (middle), and transparency (right).