Know Your Users! Estimating User Domain Knowledge in Conversational Recommenders

TL;DR

This work addresses the challenge of varying user domain knowledge in conversational recommender systems by introducing RecQuest, a gamified data-collection protocol that elicits domain-knowledge signals through target-item dialogue. It crafts a target-aware CRS architecture and a controlled experimental flow to generate dialogues across five consumer domains, accompanied by ground-truth knowledge assessments. Analyses reveal distinct interaction and linguistic patterns for novices versus experts, including differences in recommendation rate, question-asking, and attribute-focused language, suggesting the feasibility of estimating user knowledge from dialogue. The resulting RecQuest dataset offers a valuable resource for developing truly adaptive CRS that tailor elicitation and explanations to user expertise.

Abstract

The ideal conversational recommender system (CRS) acts like a savvy salesperson, adapting its language and suggestions to each user's level of expertise. However, most current systems treat all users as experts, leading to frustrating and inefficient interactions when users are unfamiliar with a domain. Systems that can adapt their conversational strategies to a user's knowledge level stand to offer a much more natural and effective experience. To make a step toward such adaptive systems, we introduce a new task: estimating user domain knowledge from conversations, enabling a CRS to better understand user needs and personalize interactions. A key obstacle to developing such adaptive systems is the lack of suitable data; to our knowledge, no existing dataset captures the conversational behaviors of users with varying levels of domain knowledge. Furthermore, in most dialogue collection protocols, users are free to express their own preferences, which tends to concentrate on popular items and well-known features, offering little insight into how novices explore or learn about unfamiliar features. To address this, we design a game-based data collection protocol that elicits varied expressions of knowledge, release the resulting dataset, and provide an initial analysis to highlight its potential for future work on user-knowledge-aware CRS.

Figures (6)

• Figure 1: An example from our collected dataset illustrating the different levels of product specification between a novice and an expert while describing their needs for a digital camera purchase. Novices tend to describe goals and contexts, whereas experts tend to specify technical constraints and components more clearly.
• Figure 2: Screenshot of the game in action, demonstrating a user who obtained several recommendations during the conversation and is ready to make a final selection.
• Figure 3: CRS Architecture.
• Figure 4: (L) Distribution of questions by fraction of participants who correctly answered them. (R) Assessed knowledge density for the three self-assessed expertise groups.
• Figure 5: (Left) Dialogue intent progression. (Right) Distribution of dialogue acts.