Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking

Taha Aksu, Nancy F. Chen

TL;DR

GCA focuses on evaluating the predicted changes in dialogue state over the entire dialogue history, and effectively reduces biases arising from distribution uniformity and the positioning of errors across turns, resulting in a more precise evaluation.

Abstract

Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit three primary limitations. They: i) erroneously presume a uniform distribution of slots throughout the dialog, ii) neglect to assign partial scores for individual turns, iii) frequently overestimate or underestimate performance by repeatedly counting the models' successful or failed predictions. To address these shortcomings, we introduce a novel metric: Granular Change Accuracy (GCA). GCA focuses on evaluating the predicted changes in dialogue state over the entire dialogue history. Benchmarking reveals that GCA effectively reduces biases arising from distribution uniformity and the positioning of errors across turns, resulting in a more precise evaluation. Notably, we find that these biases are particularly pronounced when evaluating few-shot or zero-shot trained models, becoming even more evident as the model's error rate increases. Hence, GCA offers significant promise, particularly for assessing models trained with limited resources. Our GCA implementation is a useful addition to the pool of DST metrics.

Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking

TL;DR

GCA focuses on evaluating the predicted changes in dialogue state over the entire dialogue history, and effectively reduces biases arising from distribution uniformity and the positioning of errors across turns, resulting in a more precise evaluation.

Abstract

Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit three primary limitations. They: i) erroneously presume a uniform distribution of slots throughout the dialog, ii) neglect to assign partial scores for individual turns, iii) frequently overestimate or underestimate performance by repeatedly counting the models' successful or failed predictions. To address these shortcomings, we introduce a novel metric: Granular Change Accuracy (GCA). GCA focuses on evaluating the predicted changes in dialogue state over the entire dialogue history. Benchmarking reveals that GCA effectively reduces biases arising from distribution uniformity and the positioning of errors across turns, resulting in a more precise evaluation. Notably, we find that these biases are particularly pronounced when evaluating few-shot or zero-shot trained models, becoming even more evident as the model's error rate increases. Hence, GCA offers significant promise, particularly for assessing models trained with limited resources. Our GCA implementation is a useful addition to the pool of DST metrics.
Paper Structure (29 sections, 10 equations, 8 figures, 1 algorithm)

This paper contains 29 sections, 10 equations, 8 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Related Work
  4. Joint Goal Accuracy henderson-etal-2014-second
  5. Slot Accuracy wu-etal-2021-transferable
  6. Average Goal Accuracy SGD
  7. Relative Slot Accuracy kim-etal-2022-mismatch
  8. Flexible Goal Accuracy dey-etal-2022-towards
  9. Granular Change Accuracy
  10. Value Precision:
  11. Value Recall:
  12. Label Precision:
  13. Label Recall:
  14. Experiments and Analysis
  15. Benchmarking Results
  16. ...and 14 more sections

Figures (8)

  • Figure 1: Sample task-oriented dialogue with ground truth belief state $G$, and two belief state predictions $P1$ and $P2$.
  • Figure 2: An illustrative breakdown of the four counts used in evaluating DST model predictions. Each row represents a slot label, with corresponding ground truth and model predictions. The color-coded 'Count Type' column categorizes each prediction as 'Missed', 'Wrong', 'Correct', or 'Overshot', based on the comparison between the ground truth and the predicted values.
  • Figure 3: Full and Zero-shot Results for MultiWOZ 2.1 and SGD datasets with various metrics.
  • Figure 4: \ref{['fig:T0']} and \ref{['fig:NU']} depict how spurious traits affect GCA and FGA scoring.
  • Figure 5: Variation in the absolute difference between GCA and other metrics based on the ratio of shots used during training. Red plots signify a decreasing difference, while blue plots denote an increasing difference as shots increase.
  • ...and 3 more figures