Main Predicate and Their Arguments as Explanation Signals For Intent Classification
Sameer Pimparkhede, Pushpak Bhattacharyya
TL;DR
This paper tackles the lack of ground-truth explainability in intent classification by automatically generating silver word-level explanations grounded in the main predicate and its arguments. It introduces a novel 21k-sample benchmark for ATIS and SNIPS using a dependency-based extraction method, and demonstrates that standard models excel at classification but struggle with plausibility and faithfulness in explanations. By training models with an attribution-prior loss that emphasizes these silver signals, the authors achieve 3–4% improvements in accuracy and enhanced explainability metrics, suggesting more human-like reasoning. The work offers a practical, scalable path for improving explainability in intent classification and provides a dataset and methodology that can extend to other domains and languages.
Abstract
Intent classification is crucial for conversational agents (chatbots), and deep learning models perform well in this area. However, little research has been done on the explainability of intent classification due to the absence of suitable benchmark data. Human annotation of explanation signals in text samples is time-consuming and costly. However, from inspection of data on intent classification, we see that, more often than not, the main verb denotes the action, and the direct object indicates the domain of conversation, serving as explanation signals for intent. This observation enables us to hypothesize that the main predicate in the text utterances, along with the arguments of the main predicate, can serve as explanation signals. Leveraging this, we introduce a new technique to automatically augment text samples from intent classification datasets with word-level explanations. We mark main predicates (primarily verbs) and their arguments (dependency relations) as explanation signals in benchmark intent classification datasets ATIS and SNIPS, creating a unique 21k-instance dataset for explainability. Further, we experiment with deep learning and language models. We observe that models that work well for classification do not perform well in explainability metrics like plausibility and faithfulness. We also observe that guiding models to focus on explanation signals from our dataset during training improves the plausibility Token F1 score by 3-4%, improving the model's reasoning.