BERT for Joint Intent Classification and Slot Filling
Qian Chen, Zhu Zhuo, Wen Wang
TL;DR
This work addresses data-scarce NLU for goal-oriented dialogue by fine-tuning BERT in a joint model that outputs both intents and slot labels. The approach uses the [CLS] representation for intent and first sub-token representations for slots, optionally with a CRF layer, and is trained end-to-end. It achieves significant accuracy gains on ATIS and Snips compared with prior attention-based and slot-gated models, notably improving sentence-level semantic frame accuracy. The results demonstrate strong generalization capabilities from pre-trained representations and suggest directions for applying external knowledge and broader datasets in future work.
Abstract
Intent classification and slot filling are two essential tasks for natural language understanding. They often suffer from small-scale human-labeled training data, resulting in poor generalization capability, especially for rare words. Recently a new language representation model, BERT (Bidirectional Encoder Representations from Transformers), facilitates pre-training deep bidirectional representations on large-scale unlabeled corpora, and has created state-of-the-art models for a wide variety of natural language processing tasks after simple fine-tuning. However, there has not been much effort on exploring BERT for natural language understanding. In this work, we propose a joint intent classification and slot filling model based on BERT. Experimental results demonstrate that our proposed model achieves significant improvement on intent classification accuracy, slot filling F1, and sentence-level semantic frame accuracy on several public benchmark datasets, compared to the attention-based recurrent neural network models and slot-gated models.