Chatbots language design: the influence of language variation on user experience

TL;DR

The paper investigates how linguistic register—the situationally appropriate variation in language—shapes user perceptions of chatbot interactions in tourism information contexts. By constructing parallel corpora (FLG and DailyDialog) and a modified FLG version (FLG_mod) that mimics the DailyDialog register while preserving content, the study isolates language form from content. A user study shows that register-related linguistic features more strongly predict perceived appropriateness, credibility, and overall user experience than participant or author biases, highlighting the need for register-aware and potentially dynamically adaptable language engines in chatbots. The work provides a practical methodological framework and data resources for designing and evaluating register-informed chatbots applicable to information-seeking domains beyond tourism.

Abstract

Chatbots are often designed to mimic social roles attributed to humans. However, little is known about the impact on user's perceptions of using language that fails to conform to the associated social role. Our research draws on sociolinguistic theory to investigate how a chatbot's language choices can adhere to the expected social role the agent performs within a given context. In doing so, we seek to understand whether chatbots design should account for linguistic register. This research analyzes how register differences play a role in shaping the user's perception of the human-chatbot interaction. Ultimately, we want to determine whether register-specific language influences users' perceptions and experiences with chatbots. We produced parallel corpora of conversations in the tourism domain with similar content and varying register characteristics and evaluated users' preferences of chatbot's linguistic choices in terms of appropriateness, credibility, and user experience. Our results show that register characteristics are strong predictors of user's preferences, which points to the needs of designing chatbots with register-appropriate language to improve acceptance and users' perceptions of chatbot interactions.

Figures (4)

• Figure 1: Overview of the research method. The method consists of four main steps, and the outcomes of one step is seeded into the next step.
• Figure 2: Example of a content preservation question. Participants selected their answer to the question using the slider, where 0 represents that the content in the answers is completely different, while 100 represents that the content in the answers A and B is exactly the same.
• Figure 3: Example of a question from the study. In this example, the participant was invited to select the answer that portrays the most appropriate language. Participants selected their responses by clicking on their preferred answer or on the "I don't know" option.
• Figure 4: Accuracy (a) and AUC (b) results per model for each construct (appropriateness, credibility, and user experience). The baseline represents a model that always predicts the most frequent class (original). Accuracy percentage shows that glmnet, random forest, and xgboost perform only slightly better than the baseline model. AUC, however, is reasonably better than the baseline for the three models.