Understanding Communication Preferences of Information Workers in Engagement with Text-Based Conversational Agents

TL;DR

There was a preference for relatively high empathy in wellbeing contexts and relatively low personification in coding, and similarities in preferences were also noted between applications such as chatbots for customer service and scheduling.

Abstract

Communication traits in text-based human-AI conversations play pivotal roles in shaping user experiences and perceptions of systems. With the advancement of large language models (LLMs), it is now feasible to analyze these traits at a more granular level. In this study, we explore the preferences of information workers regarding chatbot communication traits across seven applications. Participants were invited to participate in an interactive survey, which featured adjustable sliders, allowing them to adjust and express their preferences for five key communication traits: formality, personification, empathy, sociability, and humor. Our findings reveal distinct communication preferences across different applications; for instance, there was a preference for relatively high empathy in wellbeing contexts and relatively low personification in coding. Similarities in preferences were also noted between applications such as chatbots for customer service and scheduling. These insights offer crucial design guidelines for future chatbots, emphasizing the need for nuanced trait adjustments for each application.

Figures (6)

• Figure 1: Examples of chatbot responses to the same user message with different trait settings
• Figure 2: Average correlation matrix heatmap for communication traits across applications
• Figure 3: Average preference scores and their standard errors for different communication traits (x-axis) across each of the applications (colors) and the overall average (gray bar). The values inside each bar represent the corresponding average preference score for each trait.
• Figure 4: Pairwise distance matrix showing L2 distances between applications based on their average communication trait preferences.
• Figure 5: Hierarchical clustering of applications based on similarity in preferences. The y-axis represents the L2 distance, with lower branches indicating higher similarity. Clusters were formed using the average linkage method, revealing distinct groupings of similar applications.