Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systems

TL;DR

This work tackles preference elicitation in conversational recommender systems by generating implicit usage-related questions derived from item reviews. It introduces a multi-stage data annotation protocol and compares four generation models: two template-based baselines and two neural approaches (sentence-based NSQG and review-based NRQG). Automatic (BLEU-4, ROUGE-L, METEOR, accuracy) and human evaluations show neural models outperform templates, with NSQG generally preferred for quality. The offline generation pipeline and crowdsourced dataset enable scalable, interpretable question banks for real-time CRS interactions, with future work focusing on leveraging user answers and ensuring safety in language use.

Abstract

A key distinguishing feature of conversational recommender systems over traditional recommender systems is their ability to elicit user preferences using natural language. Currently, the predominant approach to preference elicitation is to ask questions directly about items or item attributes. Users searching for recommendations may not have deep knowledge of the available options in a given domain. As such, they might not be aware of key attributes or desirable values for them. However, in many settings, talking about the planned use of items does not present any difficulties, even for those that are new to a domain. In this paper, we propose a novel approach to preference elicitation by asking implicit questions based on item usage. As one of the main contributions of this work, we develop a multi-stage data annotation protocol using crowdsourcing, to create a high-quality labeled training dataset. Another main contribution is the development of four models for the question generation task: two template-based baseline models and two neural text-to-text models. The template-based models use heuristically extracted common patterns found in the training data, while the neural models use the training data to learn to generate questions automatically. Using common metrics from machine translation for automatic evaluation, we show that our approaches are effective in generating elicitation questions, even with limited training data. We further employ human evaluation for comparing the generated questions using both pointwise and pairwise evaluation designs. We find that the human evaluation results are consistent with the automatic ones, allowing us to draw conclusions about the quality of the generated questions with certainty. Finally, we provide a detailed analysis of cases where the models show their limitations.

Figures (9)

• Figure 1: Conceptual system overview. Our focus in this paper is on the implicit question generator component.
• Figure 2: Components of our template-based question generation system.
• Figure 3: Components of our neural sentence-based question generation system. The approach is similar to that of the template-based question generation, but instead of creating rigid templates, the model learns question patterns from the entire dataset automatically using a neural model.
• Figure 4: Components of our neural review-based question generation system. The model drastically simplifies inference as we do not rely on heuristics to extract candidate sentences, but take entire reviews as input to generate questions.
• Figure 5: Data collection pipeline, consisting of automatic candidate sentence extraction based on linguistic patterns and multi-step manual data annotation via crowdsourcing.