Analyzing Coherency in Facet-based Clarification Prompt Generation for Search

TL;DR

This paper investigates facet coherency as a key quality dimension for clarifying prompts in search. It defines facet coherency across semantic, syntactic, and topical axes, and develops a BART-based facet generator trained on $(q,D)$ to produce a set of facets $F$, alongside a BERT-based coherency classifier $\gamma$ trained on annotated data from the MIMICS-derived dataset. Through expert and crowdsourced annotations, the authors show that traditional NLG metrics do not reliably reflect facet coherence and that coherency interacts with the number of facets $M$ in nontrivial ways. The work provides a dataset, annotation protocols, and a predictive tool for coherency, highlighting its importance for improving user satisfaction in clarification for search.

Abstract

Clarifying user's information needs is an essential component of modern search systems. While most of the approaches for constructing clarifying prompts rely on query facets, the impact of the quality of the facets is relatively unexplored. In this work, we concentrate on facet quality through the notion of facet coherency and assess its importance for overall usefulness for clarification in search. We find that existing evaluation procedures do not account for facet coherency, as evident by the poor correlation of coherency with automated metrics. Moreover, we propose a coherency classifier and assess the prevalence of incoherent facets in a well-established dataset on clarification. Our findings can serve as motivation for future work on the topic.

Figures (1)

• Figure 1: Examples of coherent and incoherent facets for clarification in search.