Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient slot labelling

Vladimir Vlasov

TL;DR

This work tackles slot labeling in task-oriented dialogue by addressing the heavy computational burden and pretraining dependence of large PLMs. It introduces a compact architecture that uses trainable word-query attention with relative attention, a gating mechanism, and a CRF tagger, augmented by a block-diagonal dense layer to minimize parameters (approximately $10^6$). Across RESTAURANTS-8K, MTOP, and ATIS, the model matches or surpasses state-of-the-art baselines while dramatically reducing parameter count, especially in low-resource settings. The findings underscore the practicality of deploying efficient, context-aware slot-labeling models on-device, with demonstrated multilingual effectiveness and robust ablation-backed design choices.

Abstract

Slot labelling is an essential component of any dialogue system, aiming to find important arguments in every user turn. Common approaches involve large pre-trained language models (PLMs) like BERT or RoBERTa, but they face challenges such as high computational requirements and dependence on pre-training data. In this work, we propose a lightweight method which performs on par or better than the state-of-the-art PLM-based methods, while having almost 10x less trainable parameters. This makes it especially applicable for real-life industry scenarios.

Efficient slot labelling

TL;DR

This work tackles slot labeling in task-oriented dialogue by addressing the heavy computational burden and pretraining dependence of large PLMs. It introduces a compact architecture that uses trainable word-query attention with relative attention, a gating mechanism, and a CRF tagger, augmented by a block-diagonal dense layer to minimize parameters (approximately ). Across RESTAURANTS-8K, MTOP, and ATIS, the model matches or surpasses state-of-the-art baselines while dramatically reducing parameter count, especially in low-resource settings. The findings underscore the practicality of deploying efficient, context-aware slot-labeling models on-device, with demonstrated multilingual effectiveness and robust ablation-backed design choices.

Abstract

Slot labelling is an essential component of any dialogue system, aiming to find important arguments in every user turn. Common approaches involve large pre-trained language models (PLMs) like BERT or RoBERTa, but they face challenges such as high computational requirements and dependence on pre-training data. In this work, we propose a lightweight method which performs on par or better than the state-of-the-art PLM-based methods, while having almost 10x less trainable parameters. This makes it especially applicable for real-life industry scenarios.
Paper Structure (18 sections, 3 equations, 1 figure, 12 tables)

This paper contains 18 sections, 3 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Model
  5. Experimental setup
  6. Architecture
  7. Hyperparameters
  8. Datasets
  9. RESTAURANTS-8K spanconvertfewshot
  10. MTOP mtop
  11. ATIS hemphill-etal-1990-atis
  12. Baselines
  13. Results and Discussion
  14. Ablation Studies
  15. Efficiency
  16. ...and 3 more sections

Figures (1)

  • Figure 1: The scheme of the model