Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

User Intent Recognition and Satisfaction with Large Language Models: A User Study with ChatGPT

Anna Bodonhelyi, Efe Bozkir, Shuo Yang, Enkelejda Kasneci, Gjergji Kasneci

TL;DR

GPT-4 outperforms GPT-3.5 in recognizing common intents but is often outperformed by GPT-3.5 in recognizing less frequent intents, and whenever the user intent is correctly recognized, users are more satisfied with the intent-based reformulations of GPT-4 compared to GPT-3.5.

Abstract

The rapid evolution of LLMs represents an impactful paradigm shift in digital interaction and content engagement. While they encode vast amounts of human-generated knowledge and excel in processing diverse data types, they often face the challenge of accurately responding to specific user intents, leading to user dissatisfaction. Based on a fine-grained intent taxonomy and intent-based prompt reformulations, we analyze the quality of intent recognition and user satisfaction with answers from intent-based prompt reformulations of GPT-3.5 Turbo and GPT-4 Turbo models. Our study highlights the importance of human-AI interaction and underscores the need for interdisciplinary approaches to improve conversational AI systems. We show that GPT-4 outperforms GPT-3.5 in recognizing common intents but is often outperformed by GPT-3.5 in recognizing less frequent intents. Moreover, whenever the user intent is correctly recognized, while users are more satisfied with the intent-based reformulations of GPT-4 compared to GPT-3.5, they tend to be more satisfied with the models' answers to their original prompts compared to the reformulated ones. The collected data from our study has been made publicly available on GitHub (https://github.com/ConcealedIDentity/UserIntentStudy) for further research.

User Intent Recognition and Satisfaction with Large Language Models: A User Study with ChatGPT

TL;DR

GPT-4 outperforms GPT-3.5 in recognizing common intents but is often outperformed by GPT-3.5 in recognizing less frequent intents, and whenever the user intent is correctly recognized, users are more satisfied with the intent-based reformulations of GPT-4 compared to GPT-3.5.

Abstract

The rapid evolution of LLMs represents an impactful paradigm shift in digital interaction and content engagement. While they encode vast amounts of human-generated knowledge and excel in processing diverse data types, they often face the challenge of accurately responding to specific user intents, leading to user dissatisfaction. Based on a fine-grained intent taxonomy and intent-based prompt reformulations, we analyze the quality of intent recognition and user satisfaction with answers from intent-based prompt reformulations of GPT-3.5 Turbo and GPT-4 Turbo models. Our study highlights the importance of human-AI interaction and underscores the need for interdisciplinary approaches to improve conversational AI systems. We show that GPT-4 outperforms GPT-3.5 in recognizing common intents but is often outperformed by GPT-3.5 in recognizing less frequent intents. Moreover, whenever the user intent is correctly recognized, while users are more satisfied with the intent-based reformulations of GPT-4 compared to GPT-3.5, they tend to be more satisfied with the models' answers to their original prompts compared to the reformulated ones. The collected data from our study has been made publicly available on GitHub (https://github.com/ConcealedIDentity/UserIntentStudy) for further research.
Paper Structure (26 sections, 12 figures, 10 tables, 1 algorithm)

This paper contains 26 sections, 12 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Intent Recognition in User Prompts and Prompt Optimization
  3. Intent recognition in user prompts:
  4. Prompt optimization
  5. Towards a Comprehensive and Fine-grained Taxonomy of Intents in User Prompts
  6. Important Characteristics for an Effective Intent Taxonomy
  7. A Fine-Grained Intent Taxonomy
  8. User Study: Analysis of Intent Recognition and Intent-Based Prompt Reformulation
  9. User Study Phase 1: Quality of Intent Recognition by State-of-the-Art LLMs
  10. User Study Phase 2: Effect of Intent-based Prompt Reformulation on User Satisfaction
  11. Prompt Reformulation through Intent Classification
  12. The Prompt Reformulation Templates
  13. User Study Phase 3: User's Understanding of Prompt Reformulation
  14. Participant Recruitment
  15. Results and Discussion
  16. ...and 11 more sections

Figures (12)

  • Figure 1: Overview of the proposed prompt reformulation framework, visual summary of Algorithm \ref{['alg:procedure']}.
  • Figure 2: Details about the exact intent classification, represented in confusion matrices.
  • Figure 3: Data distribution corresponding to the fine-granular intent categories.
  • Figure 4: Example of the first part of the study
  • Figure 5: Example of intent prediction. For the participants, the complete fine-grained intent categories were provided.
  • ...and 7 more figures