Towards Better Understanding of User Satisfaction in Open-Domain Conversational Search

TL;DR

This work tackles open-domain ConvS evaluation by focusing on user satisfaction, identifying gaps in existing semantic-similarity and ranking-based metrics. It introduces ConvSearch, a Chinese open-domain ConvS behavior dataset with rich turn- and session-level annotations, plus a framework for third-party satisfaction annotation and revision. By comparing third-party and user annotations, it demonstrates moderate initial agreement that improves with iterative review, yet substantial divergence remains due to subjective factors. The study further proposes Dialog Continuation or Ending Behavior Modeling (DCEBM), which uses turn-level signals to predict session-level satisfaction, and shows that DCEBM can outperform traditional statistical baselines, underscoring the value of user behaviors for ConvS evaluation.

Abstract

With the increasing popularity of conversational search, how to evaluate the performance of conversational search systems has become an important question in the IR community. Existing works on conversational search evaluation can mainly be categorized into two streams: (1) constructing metrics based on semantic similarity (e.g. BLUE, METEOR and BERTScore), or (2) directly evaluating the response ranking performance of the system using traditional search methods (e.g. nDCG, RBP and nERR). However, these methods either ignore the information need of the user or ignore the mixed-initiative property of conversational search. This raises the question of how to accurately model user satisfaction in conversational search scenarios. Since explicitly asking users to provide satisfaction feedback is difficult, traditional IR studies often rely on the Cranfield paradigm (i.e., third-party annotation) and user behavior modeling to estimate user satisfaction in search. However, the feasibility and effectiveness of these two approaches have not been fully explored in conversational search. In this paper, we dive into the evaluation of conversational search from the perspective of user satisfaction. We build a novel conversational search experimental platform and construct a Chinese open-domain conversational search behavior dataset containing rich annotations and search behavior data. We also collect third-party satisfaction annotation at the session-level and turn-level, to investigate the feasibility of the Cranfield paradigm in the conversational search scenario. Experimental results show both some consistency and considerable differences between the user satisfaction annotations and third-party annotations. We also propose dialog continuation or ending behavior models (DCEBM) to capture session-level user satisfaction based on turn-level information.

Figures (7)

• Figure 1: The overall framework for data collection.
• Figure 2: Experimental platform interface of dialog webpage from user's view.
• Figure 3: Experiment procedures for the third-party user satisfaction annotations in Sec. \ref{['sec:Third-party user satisfaction annotation experiment']}
• Figure 4: The ratio of each annotation pattern among three third-party assessors in both session-level and turn-level satisfaction annotation
• Figure 5: The ratio of different types of third-party assessors merged results under AAB+ pattern and AAB- pattern