Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KULCQ: An Unsupervised Keyword-based Utterance Level Clustering Quality Metric

Pranav Guruprasad, Negar Mokhberian, Nikhil Varghese, Chandra Khatri, Amol Kelkar

TL;DR

This paper introduces Keyword-based Utterance Level Clustering Quality (KULCQ), an unsupervised metric that leverages keyword analysis to evaluate clustering quality and shows that KULCQ better captures semantic relationships in conversational data while maintaining consistency with geometric clustering principles.

Abstract

Intent discovery is crucial for both building new conversational agents and improving existing ones. While several approaches have been proposed for intent discovery, most rely on clustering to group similar utterances together. Traditional evaluation of these utterance clusters requires intent labels for each utterance, limiting scalability. Although some clustering quality metrics exist that do not require labeled data, they focus solely on cluster geometry while ignoring the linguistic nuances present in conversational transcripts. In this paper, we introduce Keyword-based Utterance Level Clustering Quality (KULCQ), an unsupervised metric that leverages keyword analysis to evaluate clustering quality. We demonstrate KULCQ's effectiveness by comparing it with existing unsupervised clustering metrics and validate its performance through comprehensive ablation studies. Our results show that KULCQ better captures semantic relationships in conversational data while maintaining consistency with geometric clustering principles.

KULCQ: An Unsupervised Keyword-based Utterance Level Clustering Quality Metric

TL;DR

This paper introduces Keyword-based Utterance Level Clustering Quality (KULCQ), an unsupervised metric that leverages keyword analysis to evaluate clustering quality and shows that KULCQ better captures semantic relationships in conversational data while maintaining consistency with geometric clustering principles.

Abstract

Intent discovery is crucial for both building new conversational agents and improving existing ones. While several approaches have been proposed for intent discovery, most rely on clustering to group similar utterances together. Traditional evaluation of these utterance clusters requires intent labels for each utterance, limiting scalability. Although some clustering quality metrics exist that do not require labeled data, they focus solely on cluster geometry while ignoring the linguistic nuances present in conversational transcripts. In this paper, we introduce Keyword-based Utterance Level Clustering Quality (KULCQ), an unsupervised metric that leverages keyword analysis to evaluate clustering quality. We demonstrate KULCQ's effectiveness by comparing it with existing unsupervised clustering metrics and validate its performance through comprehensive ablation studies. Our results show that KULCQ better captures semantic relationships in conversational data while maintaining consistency with geometric clustering principles.

Paper Structure

This paper contains 13 sections, 8 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Silhouette
  4. KULCQ
  5. Experiments and Results
  6. Datasets.
  7. Noise injection
  8. Bad Geometry, Good Semantics
  9. Penalizing high generality and variance
  10. Conclusion
  11. Datasets Description
  12. Experimental Setup
  13. Example Utterances for the Software Recommendation cluster in the AskUbuntu dataset

Figures (2)

  • Figure 1: X-axis shows the probability of perturbing each utterance's label, and the Y-axis denotes the clustering metric. We observe that both the KULCQ and Silhouette scores decrease when the probability of perturbation increases. The decrease in KULCQ score is more monotonic compared to that of the Silhouette score.
  • Figure 2: The cluster "supported cards and currencies" receives a low score from the Silhouette metric, but the KULCQ metric evaluates it as a fairly good cluster. The source of confusion for the Silhouette score is the vast difference in representation of the three subgroups of utterances. Region A includes the utterances that talk about credit card, region B includes the utterances that talk about currencies, and region C includes the utterances that mention a specific bank's name.