Analyzing and Learning from User Interactions for Search Clarification
Hamed Zamani, Bhaskar Mitra, Everest Chen, Gord Lueck, Fernando Diaz, Paul N. Bennett, Nick Craswell, Susan T. Dumais
TL;DR
The paper tackles uncertainty in search queries by analyzing how users interact with Bing's clarification pane at scale. It reveals systematic patterns in engagement, content templates, and click bias, and shows how certain query properties affect clarifications' effectiveness. To improve clarification quality, the authors introduce the RLC representation-learning framework with Intents Coverage and Answers Consistency encoders, leveraging BERT-based text encoding and attention mechanisms to re-rank candidate clarifications. Empirical results demonstrate that RLC-based re-ranking outperforms strong baselines on both click-driven and human-labeled metrics, with meaningful implications for open-domain clarification and multi-turn search interfaces.
Abstract
Asking clarifying questions in response to search queries has been recognized as a useful technique for revealing the underlying intent of the query. Clarification has applications in retrieval systems with different interfaces, from the traditional web search interfaces to the limited bandwidth interfaces as in speech-only and small screen devices. Generation and evaluation of clarifying questions have been recently studied in the literature. However, user interaction with clarifying questions is relatively unexplored. In this paper, we conduct a comprehensive study by analyzing large-scale user interactions with clarifying questions in a major web search engine. In more detail, we analyze the user engagements received by clarifying questions based on different properties of search queries, clarifying questions, and their candidate answers. We further study click bias in the data, and show that even though reading clarifying questions and candidate answers does not take significant efforts, there still exist some position and presentation biases in the data. We also propose a model for learning representation for clarifying questions based on the user interaction data as implicit feedback. The model is used for re-ranking a number of automatically generated clarifying questions for a given query. Evaluation on both click data and human labeled data demonstrates the high quality of the proposed method.