Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search

TL;DR

This paper tackles processing user responses to clarifying questions in mixed-initiative conversational search by introducing a usefulness classifier (MI-Clf) that decides whether the clarifying question, the user answer, both, or neither should be appended to the evolving query before rewriting. The method formalizes the MI CS task, selects clarifying questions via semantic similarity with a CANARD-tuned transformer, and updates the query using a four-way usefulness decision, then employs a BM25-RM3 + monoT5 + duoT5 retrieval stack. On the CASt'22 dataset, MI-Clf yields improvements over non-mixed-initiative baselines, achieving notable gains in Recall@1000 and $\text{nDCG}$ without incurring the degradations seen when always appending both questions and answers. The study also reveals that a substantial portion of interactions contain new, useful information (about 68%), while cases where neither component is helpful can degrade performance if included. Overall, MI-Clf demonstrates how targeted, usefulness-driven answer processing can enhance conversational passage retrieval and provides a blueprint for more robust MI CS systems.

Abstract

While the body of research directed towards constructing and generating clarifying questions in mixed-initiative conversational search systems is vast, research aimed at processing and comprehending users' answers to such questions is scarce. To this end, we present a simple yet effective method for processing answers to clarifying questions, moving away from previous work that simply appends answers to the original query and thus potentially degrades retrieval performance. Specifically, we propose a classifier for assessing usefulness of the prompted clarifying question and an answer given by the user. Useful questions or answers are further appended to the conversation history and passed to a transformer-based query rewriting module. Results demonstrate significant improvements over strong non-mixed-initiative baselines. Furthermore, the proposed approach mitigates the performance drops when non useful questions and answers are utilized.