A Two-dimensional Zero-shot Dialogue State Tracking Evaluation Method using GPT-4

TL;DR

This work addresses the limitations of traditional DST evaluation that rely on labeled data and exact-match criteria, which can misrepresent semantic correctness. It presents a two-dimensional zero-shot evaluation framework using GPT-4 that separately analyzes accuracy and completeness of predicted DST states, aided by manually designed reasoning prompts, and integrates results to produce per-turn assessments and $JGA$-like coherence without annotations. The method demonstrates competitive Turn State Accuracy and reduces over-evaluation compared to string-match baselines, while showing consistency with conventional evaluation results across datasets. Overall, the approach provides a data-efficient, semantically aware evaluation paradigm for DST that can generalize beyond annotated benchmarks and informs model development and benchmarking.

Abstract

Dialogue state tracking (DST) is evaluated by exact matching methods, which rely on large amounts of labeled data and ignore semantic consistency, leading to over-evaluation. Currently, leveraging large language models (LLM) in evaluating natural language processing tasks has achieved promising results. However, using LLM for DST evaluation is still under explored. In this paper, we propose a two-dimensional zero-shot evaluation method for DST using GPT-4, which divides the evaluation into two dimensions: accuracy and completeness. Furthermore, we also design two manual reasoning paths in prompting to further improve the accuracy of evaluation. Experimental results show that our method achieves better performance compared to the baselines, and is consistent with traditional exact matching based methods.

Figures (3)

• Figure 1: The framework of our proposed evaluation method for dialogue state tracking.
• Figure 2: Turn state accuracy evaluated by different methods.
• Figure 3: Joint goal accuracy evaluated by different methods.