CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems

TL;DR

This work tackles the robustness gap in user satisfaction estimation (USE) for task-oriented dialogue (TOD) systems under imbalanced label distributions. It introduces satisfaction-focused counterfactual dialogue generation using open-source LLMs, with human validation, to augment test collections for the MultiWOZ and SGD benchmarks. Through extensive experiments, the authors show that few-shot in-context learning (ICL) with LLMs yields higher robustness to increasing dissatisfaction samples than fine-tuned baselines, particularly on the augmented CF and Mix test sets. They also demonstrate high-quality data via human annotation (coherence and satisfaction) and release the aligned counterfactual dialogues to support further research. This approach highlights the value of data augmentation for USE in TODs and points to broader adoption of LLM-generated counterfactual data for robust evaluation and model development.

Abstract

An important unexplored aspect in previous work on user satisfaction estimation for Task-Oriented Dialogue (TOD) systems is their evaluation in terms of robustness for the identification of user dissatisfaction: current benchmarks for user satisfaction estimation in TOD systems are highly skewed towards dialogues for which the user is satisfied. The effect of having a more balanced set of satisfaction labels on performance is unknown. However, balancing the data with more dissatisfactory dialogue samples requires further data collection and human annotation, which is costly and time-consuming. In this work, we leverage large language models (LLMs) and unlock their ability to generate satisfaction-aware counterfactual dialogues to augment the set of original dialogues of a test collection. We gather human annotations to ensure the reliability of the generated samples. We evaluate two open-source LLMs as user satisfaction estimators on our augmented collection against state-of-the-art fine-tuned models. Our experiments show that when used as few-shot user satisfaction estimators, open-source LLMs show higher robustness to the increase in the number of dissatisfaction labels in the test collection than the fine-tuned state-of-the-art models. Our results shed light on the need for data augmentation approaches for user satisfaction estimation in TOD systems. We release our aligned counterfactual dialogues, which are curated by human annotation, to facilitate further research on this topic.

Figures (7)

• Figure 1: Example dialogue (snippet) between the user and the system from the MultiWOZ benchmark.
• Figure 2: Examples of generated counterfactual system utterances. Dissatisfaction to Satisfaction (left) and vice versa (right). See Figure \ref{['fig:full_counterfactual_samples']} in the Appendix for the full dialogues corresponding to these examples.
• Figure 3: The input used as the prompt for LLMs in order to predict the user satisfaction label.
• Figure 4: Performance of USE models with a varying degree of imbalance in the test set for the MultiWOZ and SGD benchmarks. The dissatisfaction ratio is the proportion of samples with dissatisfaction labels in the test collection.
• Figure 5: Sensitivity of the models in identification of user dissatisfaction on various proportions of dissatisfaction test samples.