Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChatGPT Role-play Dataset: Analysis of User Motives and Model Naturalness

Yufei Tao, Ameeta Agrawal, Judit Dombi, Tetyana Sydorenko, Jung In Lee

TL;DR

This study addresses the gap in how ChatGPT behaves during conversations in different settings by analyzing its interactions in both a normal way and a role-play setting, and introduces a novel dataset of broad range of human-AI conversations annotated with user motives and model naturalness.

Abstract

Recent advances in interactive large language models like ChatGPT have revolutionized various domains; however, their behavior in natural and role-play conversation settings remains underexplored. In our study, we address this gap by deeply investigating how ChatGPT behaves during conversations in different settings by analyzing its interactions in both a normal way and a role-play setting. We introduce a novel dataset of broad range of human-AI conversations annotated with user motives and model naturalness to examine (i) how humans engage with the conversational AI model, and (ii) how natural are AI model responses. Our study highlights the diversity of user motives when interacting with ChatGPT and variable AI naturalness, showing not only the nuanced dynamics of natural conversations between humans and AI, but also providing new avenues for improving the effectiveness of human-AI communication.

ChatGPT Role-play Dataset: Analysis of User Motives and Model Naturalness

TL;DR

This study addresses the gap in how ChatGPT behaves during conversations in different settings by analyzing its interactions in both a normal way and a role-play setting, and introduces a novel dataset of broad range of human-AI conversations annotated with user motives and model naturalness.

Abstract

Recent advances in interactive large language models like ChatGPT have revolutionized various domains; however, their behavior in natural and role-play conversation settings remains underexplored. In our study, we address this gap by deeply investigating how ChatGPT behaves during conversations in different settings by analyzing its interactions in both a normal way and a role-play setting. We introduce a novel dataset of broad range of human-AI conversations annotated with user motives and model naturalness to examine (i) how humans engage with the conversational AI model, and (ii) how natural are AI model responses. Our study highlights the diversity of user motives when interacting with ChatGPT and variable AI naturalness, showing not only the nuanced dynamics of natural conversations between humans and AI, but also providing new avenues for improving the effectiveness of human-AI communication.
Paper Structure (13 sections, 7 figures, 3 tables)

This paper contains 13 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CRD: ChatGPT Role-play Dataset
  4. Data Collection
  5. Data Annotation
  6. Research Questions
  7. Analysis and Results
  8. Statistical Analysis
  9. User Motives and Model Naturalness
  10. Perplexity and Sentiment Analysis
  11. Discussion and Future Work
  12. Conclusion
  13. Bibliographical References

Figures (7)

  • Figure 1: Snippets of conversations from our dataset CRD, where vanilla, boss, and classmate denote the three subsets of CRD. 'H' denotes the human utterance, whereas 'C' indicates the response generated by ChatGPT.
  • Figure 2: (A1 and A2) Schematic differences between conversations in vanilla and role-play datasets (boss/classmate)
  • Figure 3: (A1) Unique number of topic words and their overlap across three datasets
  • Figure 4: (A3, A6 and A7) Distribution of user motives (top) and model naturalness (bottom)
  • Figure 5: (A8) Percentage of follow-up model naturalness categories after each user motive in vanilla (top), boss (middle) and classmate (bottom). Rows represent user motives, columns indicate naturalness categories, and shading intensity signifies percentage occurrences.
  • ...and 2 more figures