Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unveiling the Secrets of Engaging Conversations: Factors that Keep Users Hooked on Role-Playing Dialog Agents

Shuai Zhang, Yu Lu, Junwen Liu, Jia Yu, Huachuan Qiu, Yuming Yan, Zhenzhong Lan

TL;DR

This study investigates why users stay engaged in long-running conversations with role-playing dialog agents using real-world interaction data. It defines and quantifies nine potential retention factors and applies a permutation-based significance test across 53 strong–weak model pairs to identify influential cues. The results show that turn length and certain communicative cues (non-verbal description, human-likeness, personality consistency) significantly boost retention, while factors like embodiment, diversity, empathy, and proactivity have limited impact. These findings provide concrete design guidance for developing role-playing large language models and highlight the importance of narrative depth and character portrayal in sustaining long-term user engagement.

Abstract

With the growing humanlike nature of dialog agents, people are now engaging in extended conversations that can stretch from brief moments to substantial periods of time. Understanding the factors that contribute to sustaining these interactions is crucial, yet existing studies primarily focusing on short-term simulations that rarely explore such prolonged and real conversations. In this paper, we investigate the factors influencing retention rates in real interactions with roleplaying models. By analyzing a large dataset of interactions between real users and thousands of characters, we systematically examine multiple factors and assess their impact on user retention rate. Surprisingly, we find that the degree to which the bot embodies the roles it plays has limited influence on retention rates, while the length of each turn it speaks significantly affects retention rates. This study sheds light on the critical aspects of user engagement with role-playing models and provides valuable insights for future improvements in the development of large language models for role-playing purposes.

Unveiling the Secrets of Engaging Conversations: Factors that Keep Users Hooked on Role-Playing Dialog Agents

TL;DR

This study investigates why users stay engaged in long-running conversations with role-playing dialog agents using real-world interaction data. It defines and quantifies nine potential retention factors and applies a permutation-based significance test across 53 strong–weak model pairs to identify influential cues. The results show that turn length and certain communicative cues (non-verbal description, human-likeness, personality consistency) significantly boost retention, while factors like embodiment, diversity, empathy, and proactivity have limited impact. These findings provide concrete design guidance for developing role-playing large language models and highlight the importance of narrative depth and character portrayal in sustaining long-term user engagement.

Abstract

With the growing humanlike nature of dialog agents, people are now engaging in extended conversations that can stretch from brief moments to substantial periods of time. Understanding the factors that contribute to sustaining these interactions is crucial, yet existing studies primarily focusing on short-term simulations that rarely explore such prolonged and real conversations. In this paper, we investigate the factors influencing retention rates in real interactions with roleplaying models. By analyzing a large dataset of interactions between real users and thousands of characters, we systematically examine multiple factors and assess their impact on user retention rate. Surprisingly, we find that the degree to which the bot embodies the roles it plays has limited influence on retention rates, while the length of each turn it speaks significantly affects retention rates. This study sheds light on the critical aspects of user engagement with role-playing models and provides valuable insights for future improvements in the development of large language models for role-playing purposes.
Paper Structure (12 sections, 12 equations, 3 figures, 11 tables)

This paper contains 12 sections, 12 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Factors Definition
  5. Significance Test
  6. Experimental results
  7. Setup
  8. Overall results
  9. Factor Analysis
  10. Conclusion
  11. Limitation
  12. Exemplar dialogs

Figures (3)

  • Figure 1: Our method follows a pipeline that begins by identifying pairs of (strong and weak) models with significantly different retention rates through A/B testing. From these pairs, we then sample dialog examples and utilize them to calculate scores for various factors. Finally, we quantify the impact of these factors and their significance by analyzing the calculated scores.
  • Figure 2: Comparison of factor scores between strong and weak models is visualized in a grid of sub-figures. Each point in the grid represents a pair of strong and weak models, with the x-axis and y-axis denoting the factor scores of the strong and weak models, respectively. A dotted line within each sub-figure indicates points with identical factor scores between the model pairs. More points located towards the top-left (bottom-right), away from the dotted line, suggest a stronger positive (negative) influence on the retention rates.
  • Figure 3: This figure shows the z-scores of each factor. Blue bars represent factors with positive z-scores, while orange bars with dashed outlines represent factors with negative z-scores. The presence of a check marker on top of the bars indicates that these factors have a statistically significant influence on retention rates (p-value < 0.05).