Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions

TL;DR

This work investigates how the alignment of AI agent expertise with user domain knowledge affects user experience in multi-domain Copilot interactions. By building a 5-point ordinal expertise classifier and labeling three expertise dimensions (User, Gauged User, Agent) via GPT-4 prompting with human validation, the study analyzes 25,033 conversations to quantify misalignment effects. Key findings show the agent is typically at proficient or expert levels (76–77%) and that misalignment—especially when users are more expert than the agent—reduces satisfaction, with stronger effects on high-complexity tasks; engagement tends to rise when the agent’s expertise matches the user’s. The results highlight the importance of expertise alignment in designing human-centered AI and suggest future work on interventions to balance high, broadly capable AI with user-specific tuning, while acknowledging correlational limits and multilingual/generalizability considerations.

Abstract

Using a sample of 25,000 Bing Copilot conversations, we study how the agent responds to users of varying levels of domain expertise and the resulting impact on user experience along multiple dimensions. Our findings show that across a variety of topical domains, the agent largely responds at proficient or expert levels of expertise (77% of conversations) which correlates with positive user experience regardless of the user's level of expertise. Misalignment, such that the agent responds at a level of expertise below that of the user, has a negative impact on overall user experience, with the impact more profound for more complex tasks. We also show that users engage more, as measured by the number of words in the conversation, when the agent responds at a level of expertise commensurate with that of the user. Our findings underscore the importance of alignment between user and AI when designing human-centered AI systems, to ensure satisfactory and productive interactions.

Figures (11)

• Figure 1: An overview of our expertise classifier pipeline.
• Figure 2: Barplots showing the distribution of User Expertise (left) and Agent Expertise (right) on different domains of Copilot conversations.
• Figure 3: Piecewise regression plots showing the correlation between Expertise Difference and SAT scores.
• Figure 4: Barplot showing the distribution of Gauged User Expertise on different domains of Copilot conversations.
• Figure 5: Heatmaps between User and AI expertise (left) and User and Gauged User expertise (right).